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ABSTRACT 

The House Subcommittee on Select Education asked the 
General Accounting Office (GAO) to study the condition of information 
on education in the United States during selected years between 1973 
and 1986. The resulting report covers three areas: (1) the nature and 
amount of federally sponsored information on education; (2) the 
quality of such information; and (3) factors influencing production 
and quality of the information. The study consisted of a review of 
information-gathering activities of the three principal units of the 
Department of Education, namely, the National Institute of Education; 
the National Center for Education Statistics; and the Office of 
Planning, Budget, and Evaluation. The GAO also examined the quality 
of information through analysis of three statistical programs, 
namely, the National Assessment of Educational Progress, the Common 
Core of Data for Elementary and Secondary Education, and the Fast 
Response Survey System. Performance assessments concentrated on 
relevance r timeliness, technical adequacy, and impact. Results 
indicate c decline in federally sponsored research and in statistical 
and evaluative information on education; a shift away from collection 
of new data to service-oriented activities; and a trend toward 
contract awards to institutions rather than field-related endeavors. 
Agency comments and recommendations are outlined. (TJH) 
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GAO 



United SUtes 

Generml Accounting Office 

Washinirton, D.C. 20548 



Program Evalnatton and 
Methodology Division 

B-226398 

November 4, 1987 

The Honorable M£Uor Owens 
Chairman, Subcommittee on Select Education 
Committee on Education and Labor 
House of Representatives 

Dear Mr. Chairman: 

At the Subcommittee's request, we examined the condition of information on education in the 
United States. As was agreed with your office, we focused on three features of information 
stemming from federally supported educational research, statistics, and evaluation. First, we 
examined the production of selected types of educational information from the early 1970's 
to 1986. Specifically, we looked at the changes in iiiformation activities, priorities, and 
participants. Second, we reviewed three statistical programs — the National Assessment of 
Educational Progress, Common Core of Data for elementary and secondary education, and 
Fast Response Survey System — to determine how the quality of these programs has 
changed. Third, we identified key factors influencing information production and quality. 

As arranged with your office, unless you publicly announce the contaits of this report 
earlier, we plan no further distribution of it until 30 days from the date of the report. At that 
time, we will send copies to the Department of Education and others who are interested and 
make copies available to others upon request. 

Sincerely yours. 



Eleanor Chelimsky 
Director 
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PllfTirmP Although education in this country is a responsibility of the states, the 

^ federal government spends nearly $20 billion annually to support all 

levels of educacion. Further, since the Congress first authorized a non- 
cabinet department of education in 1867, the federal government has 
maintained a strong interest in who is getting educated and what they 
are learning. The Congress and the education community have, however, 
expressed concern about how well the department is carrying out its 
information-gathering function. !n light of these concerns, the House 
Subcommittee on Select Education asked gag to study the condition of 
information on education in the United States. This report addresses 
three questions: (1) What federally sponsored information on education 
is being produced and how has it changed? (2) What is the quality of the 
information and how has the quality changed? (3) What factors influ- 
ence the production and quality of information? 



GAG examined information production by reviewing the information- 
gathering activities of the three principal units in the Department of 
Education responsible for education information during the time of our 
review: the National Institute of Education (nie), the National Center for 
Education Statistics (nces), and the Office of Planning, Budget, and 
Evaluation (gpbe). gag examined the quality of information through 
analyses of three statistical programs: the National Assessment of Edu- 
cational Progress, the Common Core of Data for elementary and second- 
ary education, and the Fzst Response Survey System, gag assessed 
performance in terms of four indicators of quality: relevance, timeliness, 
technical adequacy, and impact. From the results of these reviews, 
together with an examination of relevant documents and interviews, 
GAG identified factors influencing information production and quality. 
gag's review covers selected years between 1973 and 1986. 



R6SUltS in Bri6f During the past decade, the production of federally sponsored research, 

statistical, and evaluative information on education has declined nota- 
bly. Research and evaluation activities were hardest hit in terms of 
reductions in number of awards between 1980 and 1985. Research activ- 
ities shifted away from the collection of new data to service-oriented 
activities such as dissemination, so much so that the availability of up- 
to-date information to disseminate to teachers and other practitioners 
may be threatened. Further, the new data collection efforts that were 
undertaken during the period of this review increasingly became more 
narrowly focused and the scope of investigation was also restricted by 
increased use of contracts awarded to institutions rather than field-initi- 
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ated grants. While some high-quality statistical information was 
being produced, the quality was variable. In two programs, quality 
was maintained or improved on some dimensions but in the Common 
Core of Data, data quality problems have persisted for several 
decades. The major influence on information production was severe 
reductions in funding levels. Activities that did not carry congres- 
sional mandates were most vulnerable to funding declines and 
changes in priorities, which also were linked to rapid changes in 
leadership. Expert review of specific information-gathering activi- 
ties had a positive influence on quality in some instances. Results 
were clearest when several of these factors coexisted and worked in 
the same direction. 



Principal Findings 



Information Production The number of grants and contracts awarded for research decreased 65 

percent from 476 in 1980 to 168 in 1985. The number of evaluation con- 
tracts peaked at 1 19 in 1980 and progressively dropped 79 percent to 25 
in 1985. Statistical surveys, planned or conducted, fell 31 percent 
between 1980 and 1983 from 55 to 38. The intervals between data col- 
lections increased and technical support to the states for data collection 
was sharply reduced. (See pages 20-24.) 

The information that was produced by awards also changed. Sixty-five 
percent of nie's 1980 awards but only 1 1 percent of the 1985 awards 
were for new data collection. Awards for service activities such as dis- 
seminating information and providing expert testimony in civil rights 
cases increased from 35 percent to 89 percent of all awards. Fewer edu- 
cational areas were investigated in 1985 than in 1980 through research 
grants. In 1980, for example, 56 of 293 awards for new data collection 
went towai J studies of special populations such as minorities and 
women. In 1985, there were five such studies. Some areas such as learn- 
ing in nonschool settings and areas identified as **school problems" 
(including such issues as dropouts and delinquency) received no new 
data collection funds at aU in 1985; in 1980, there were 33 awards. (See 
pages 31-36.) 

Further, those who carried out the work shifted. The proportion of 
research awards made to department-sponsored institutions (for exam- 
ple, laboratories and national centers) increased substantially from 1980 
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to 1985. In 1980, institutions received 25 percent of the awards in three 
m^jor program areas, compared to 56 percent in 1985. In 1980, 23 per- 
cent of nie's awards were made through contracts; in 1985, 86 percent. 
OPBE funded nearly all 1985 evaluations through competitively awarded 
contracts; in 1980, the award process was more diverse. Thus informa- 
tion-gathering was increasingly more likely to be prescribed by the 
agency than to have been proposed from the field. (See pages 36-38.) 



A review of relevance, timeliness, technical adequacy, and impact shows 
that quality varied. The National Assessment of Educational Progress 
received generally high marks, although efforts to optimize one aspect 
of quality were associated with losses in other dimensions. The Fast 
Response Survey System received relatively high marks on relevance 
and medium ratings on technical adequacy and timeliness. However, the 
Common C!ore of Data, adequate in some respects, was generally poor in 
its quality of information. Many problems — some of which had been 
identified by others several decades earlier — remain. (See chapter 3.) 



Influences on Information Support for research has decreased since the early 1970's by more than 
Production and Quality ^® percent in constant dollars, despite the fact that the federal invest- 
ment in education increased by 38 percent and federal support for 
research in general increased by about 4 percent in constant dollars 
between 1980 and 1984. Funding for statistics and evaluation also 
declined more than in these areas for the government in general. The 
patterns of fiscal declines in research, evaluation, and statistical activity 
corresponded to reductions in information production. (See pages 68- 
72.) 

Although all information-gathering activities were affected by budget 
constraints, congressionally mandated activities received smaller reduc- 
tions and thereby consumed an increasing share of available resources. 
Activities that were not required by law were vulnerable to changes in 
priorities, fimding, and policies. Rapid turnover of top leadership, espe- 
cially in NiE, was associated with decisions not to fund areas of research 
initiated under other directors. (See pages 76-78 and 83.) 

In the three statistical programs, relevance was increased by adding 
data elements, tailoring data collection to the needs of specific reques- 
ters, and making dissemination flexible. Timeliness was improved by 
releasing data early and diversifjring their formats. Technical adequacy 
was higher for surveys than for data from state administrative records. 



Quality of Statistical 
Programs 
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Some information-gathering activities reviewed comprehensively by 
technical experts improved in quality. (See pages GO-66.) 



Recommendations 



GAG does not present recommendations in this report. 



Agency Comments 



The Department of Education generally agreed with gag's findings, stat- 
ing that the report will perform a valuable function in documenting a 
long-term decline in resources for educational information. However, the 
department expressed three concerns about the report. First, citing the 
many organizational changes initiated since 1985, the department 
believed gag's analyses did not reflect the current situation. Second, it 
questioned gag's asser sment of shifts in priorities, stating that greater 
emphasis on dissemination represented a positive step, changes in lead- 
ership did not affect research pnorities, and almost all important areas 
were being investigated through a variety of strategies. Third, the 
department took issue with gag's analysis of the implications of changes 
in who is producing information and how it is funded. Department offi- 
cials also provided detailed descriptions of recent organizational 
changes, docmnentation on budgets and activities not covered in gag's 
review, and further specific conunents. (See appendix IV.) 

GAG acknowledges the numerous changes since fiscal year 1986. How- 
ever, it is too early to determine whether these changes will adequately 
address the problems identified in this report or the new problems that 
the changes themselves might create. Empirical assessment of the pro- 
duction and quality of information will be necessary. 

With regard to shifts in priorities, gag maintains that dissemination can 
remain a critical part of the research process only if the data that are 
being disseminated are relevant and timely, gag continues to conclude 
that changes in leadership did affect priorities and notes that while 
information is being collected on contemporary problems, the depart- 
ment seems to lack formal mechanisms for identifying emerging issues. 

GAG continues to believe that while contracts provide a needed basis for 
accountability, widespread use of contracts has other, less positive con- 
sequences. For example, requests for proposals often specify the scope 
of work, leaving little flexibility for the imaginative researcher. While 
GAG commends the department's efforts to restore some of the avenues 
for new data collection — such as the unsolicited-grants program — cur- 
rent levels of support are dramatically lower than in 1980. 
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Introduction 



Although education is a responsibility of the states, there is a strong and 
abiding national interest in who is getting educated and what they are 
learning. Since 1867, the Congress has authorized the Department of 
Education to obtain information on the condition of education for pur- 
poses of identifying emerging needs, determining how well programs are 
working, and promoting educational improvement. At the request of the 
Subcommittee on Select Education of the House Committee on Education 
and Labor, we examined the condition of this information about educa- 
tion: that is, how and how well the information-gathering function has 
been carried out and what more could be done to improve the produc- 
tion and quality of information. 



The Federal Role in 
Education Infonnation 



Although the federal government currently spends nearly $20 billion 
annually to support education, its earliest task was gathering informa- 
tion. Federal involvement in education information dates back to 1867, 
when the Congress created a noncabinet Department of Education.* The 
department's initial mandate was to gather statistics on U.S. education. 
Although the federal role in education has changed during the many 
years since the department's inception, information-gathering has 
T-emained one of its important functions. Information-gathering units 
have expanded their scope beyond gathering statistics. Their work now 
includes, for example, the sponsorship of research and evaluating the 
educational programs administered by the Department of Education as 
well as technical assistance and dissemination. 



Roles and Responsibilities 
of Information-Producing 
Units 



During the 1970's, the responsibility for research, statistics, and evalua- 
tion were assigned to the National Institute of Education (nie), the 
National Center for Education Statistics (nces), and the Office of Plan- 
ning, Budget, and Evaluation (opbe), respectively. There was some over- 
lap of activities (for example, nie and opbe have conducted evaluations 
of programs), but these units have had fairly distinct information-gath- 
ering roles within the department. Their origins and missions are 



^Although the Department of EducaUon was not made a cabinet department until 1979, we refer to it 
as the Department of Education. 
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described below. We also highlight how these missions have changed 
over time.2 



The National Institute of The National Institute of Education was originally created in the Educa- 

Education tion Amendments of 1972. In establishing nie, the Congress declared 

that to provide high quality education, **far more dependable knowledge 
about the processes of learning and education than now exists or can be 
expected from present research and experimentation" was required. 
The legislation charged nie with the responsibility of building **an effec- 
tive educational research and de\e^opment system." 

nie' J mission as given in the public law was fairly general, leaving con- 
siderable flexibility in the development of the research function and 
specific areas of focus to its director and policymaking board, the 
National Council on Educational Research (ncer).^ In subsequent years, 
the Congress used legislation to indicate its priorities for the National 
Institute of Education. For example, listed among the priorities in the 
Educational Amendments of 1976 were improving student achievement 
in the basic skills, including reading and mathematics; improving the 
ability of schools to meet their responsibilities to provide equal educa- 
tional opportunities, including students who are socially, economically, 
or educationally disadvantaged; and improving dissemination of the 
results of, and knowledge gained from, educational research and devel- 
opment. In addition to providing priority areas during the 1970's, the 



^In the department's October 1985 reorganization, NIE and NOES were discontinued as separate 
agencies and all their functions and activities were assigned to the five operating units of the Office 
of Educational Research and Improvement (OERI). (The five units are Office of Research, Center for 
Education Statistics (CES^ Pro^tuns for the Improvement of Practice, Information Services, and 
Library Programs.) The Center for Education Statistics performs most of the former responsibilities 
of NCES. And although some NIE responsibilities have been transferred to the new units, the Office 
of Research now carries out the activities of NIE that we discuss in this report. Because our review 
covers the period prior to the departmental reorganization, we refer to each unit by its name applica- 
ble during that period— that is, NIE, NCES, and OPBE. 

^ince the reorganization, NCER has been renamed the National Advisory Council on Educational 
Research and Improvement Although its purview has been expanded to include all activities m the 
Office of Educational Research and Improvement, its role has been changed from policy to advisory. 
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Congress increasingly required nie to conduct specific studies, evalua- 
tions, and activities (for example, support for regional educational labo- 
ratories, national research centers, and the National Assessment of 
Educational Progress).^ 



The National Center for The statistical activities that had been performed in the department 

Education Statistics since its early days were organized into the National Center for Educa- 

tion Statistics in 1974 (Public Law 93-380).^ As stated in the Education 
Amendments of 1974, 

"the purpose of the Center shall be to collect and disseminate statistics and other 
data related to education in the United States and in other nations. The Center shall 
. . . collect, collate, and, from time to time, report full and complete statistics on the 
condition of education in the United States; conduct and publish reports on special- 
ized analyses of the meaning and significance of such statistics; . . . (and) review and 
report on education activities in foreign countries." 

NCES was authorized to produce statistical data, but, in general, the tjrpe 
of information that was to be collected, and when and how it was to be 
collected were not initially specified by the Congress. Over time, the 
Congress has amended the mission of nces by adding requirements for 
assistance to state and local education agencies to improve their statisti- 
cal and data collection activities. Sev( ^1 special and recurring surveys 
were also mandated at various points over the past decade (for example, 
a survey of institutions of higher education and teacher demand-and- 
shortage studies). 

Office of Planning, Budget, and In 1970, the Office of Planning, Budget, and Evaluation became the 
Evaluation department's central office for program evaluation activities. Although 

OPBE is not authorized by legislation, its responsibilities have included 



^The labOTatories conduct studies, disseminate research findings, and provide technical assistance to 
educational instituti<HW in their assigned geographic regions; the national centers conduct research on 
the topics or issues they have received awards to study (for example, teaching, reading, and voca- 
tional education). Some natiorial centers have been supported by NIE discretionary funds. Most of the 
support for the centers and ail the support far the laboratories has been congressionally mandated, 
however. We refer to them collectively as the laboratories and centers. For some analyses, however, 
we discuss the mandated and discretionary laboratories and centers separately. In the recent reorgan- 
ization, some mandated activities including the laboratories and the educational information dissemi- 
nation centers were transferred from NIE to a newly created unit responsible for the improvement of 
educational practice. We discuss laboratories and centers further in chapter 2. 

^NCES was made a statutory entity in 1974, and it was established ad' ttrativdy in January 1966 
as a staff office reporting directly to the commissioner of education. 

!1 
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^^implementation of Congressional mandates, conduct of program impact studies, 
meeting information needs of OE (Office of Education] program managers, provision 
of technical assistance in evaluation to the field, studies to identify effective pro- 
gram services, and practices or projects for improvement of program operations and 
for dissemination to the field/* 

Although a number of evaluations are carried out in other department 
units, OPBE is considered the central location for departmental evalua- 
tions. In addition to its earlier roles and responsibilities, opbe currently 
reviews proposed legislation, regulations, and administrative orders or 
public announcements that affect policy, program plans, and budgets. 
OPBE has developed and monitored the secretary's policy agenda, 
although this was last done in 1983. opbe brings together the results of 
research, analysis, plaiming, implementation, and evaluation activities 
of all the principal offices of the department. 

Although evaluation is clearly opbe's charge, its roles and responsibili- 
ties have shifted in recent years. In an August 20, 1986, memo, the dep- 
uty undersecretary for planning, budget, and evaluation established 
procedures for planning and coordinating evaluation studies within the 
department. In implementing this policy, the memo outlined three activi- 
ties, including the preparation of an inventory of all current evaluation 
studies, the preparation of an evaluation plan for future studies, and the 
review of work statements for procurements to ensure policy relevance 
and methodological adequacy, opbe approval is required for all evalua- 
tion plans and work statements. 



Concern Over Department 
Performance 



Cioncem over how the department is performing its information nussion 
goes back to 1869. Indeed, the first conunissioner of education resigned 
under pressure after roughly 3 years in office for failing to provide 
quality information quickly enough. More recently, criticism has been 
expressed in congressional hearings and conference reports. At the 
request of various congressional committees, gao has in the past investi- 
gated such aspects of the department's performance as its evaluation of 
compensatory education programs, its management of the National 
Assessment of Educational Progress (naep), and its adherence to 
requirements of impartiality in awarding research grants and contracts. 
While there is general agreement that support for information-gathering 
is an appropriate and necessary federal role, there also has been contin- 
ued unease with the department's performance of its information-gath- 
ering functions. 
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We addressed the Subcommittee's concern about the condition of educa- 
tion information by organizing our review around three questions. 

1. What federally sponsored information on education is being produced 
and how has it changed over time? 

2. What is the quality of the information and how has quality changed 
over time? 

3. What fa^ jots influence information production and quality? 

In answering these questions, we focused on research, statistical, and 
evaluative information. As described above, our data collection efforts 
were limited to three organizations — the National Institute of Education, 
the National Center for Education Statistics, and the Office of Planning, 
Budget, and Evaluation. 

While information is also gathered by numerous other units within the 
department (the Office of Civil Rights and the Office of Special Educa- 
tion programs, among others), the organizations we focused on are 
responsible for the msyority of the information-producing activities. 

We looked at information that was produced from the early 1970's to 
1985. We chose these years because they allowed us to compare infor- 
mation production in the early days of each organization with informa- 
tion production in more recent years. (Although 1985 was the last year 
for which reasonably complete information was available, in a few anal- 
yses we were also able to review 1986 activities.) In addition, the longi- 
tudinal i>erspective allowed us to look at the information production 
process, permitting us to follow some information activities from imple- 
mentation to use. 



Study Plan Our plan for data collection had several components. First, to describe 

the information that was produced and to document changes in that 
information over time, we reviewed the activities the organizations had 
been involved in since the early I970*s. We attempted to identify infor- 
mation about contract and grant awards and in-house work the organi- 
zations had performed. In particular, we analyzed agency publications 
and data on awards. We used number and type of awarcte for opbe and 
NiE and number and type of surveys for noes as indicators of informa- 
tion that was produced through the department. 

^6 
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These are not direct measures of the information that was produced. 
However, given the time it takes to complete research and the many pos- 
sible channels of its release, it is difficult to measure information output 
adequately. We believe that measures of new and ongoing work serve as 
appropriate indicators of information production. Where possible, we 
also examined published reports. 

To answer the question on quality, we reviewed the available literature 
on quality and we developed case examples of three major information- 
gathering activities. For each case, we drew on our own analyses and on 
prior reviews by experts as assessments of the activity's technical qual- 
ity. Since the Subconunittee was interested in the effects of different 
ways 0"' requesting information, such as the use of mandates and discre- 
tionary funds, we deliberately chose cases that represented a variety of 
such requests. 

To examine factors affecting information production and quality, we 
brought together findings from the three case examples, other case anal- 
yses, interviews with agency officials, and analyses of agency docu- 
ments. We analyzed the individual budgets and the budget of the 
Department of Education as a whole. We obtained these data from 
agency officials and publications. We also drew on the literature of each 
of the organizations and on education information, information quality, 
and request strategies. Table 1.1 links our study questions with our data 
collection plan. 



Table 1.1: Our Study Questions and Data 
We Collected for Them 



Question 


Source 


1 . What federally sponsored information on 


~ Agency publications and data 


education is being produced and how has 


It changed over time? 




2 What IS the quality of the information and 


~ Case studies 


how has quality changed over time'^ 


~ Literature on quality 



3 What factors influence information ~ Budgets 

production and quality*? — Case studies and other examples 

— Interviews with agency officials and 

agency documents 
~ Primary and secondary literature on the 
agency, education data and their quality, 
and request strategies 



Obtaining Information on We were interested in ar1;ivities from the early 1970's to 1985 and, at 
Elducation minimum, we sought to identify who (individuals versus institu- 

tions) performed each activity and starting and completion dates, areas 
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of concern, and funding levels. None of the orgariizations had an infor- 
mation management system that could give us complete information. 
Complete data were available for a few years at most; for other years, 
and especially before 1980, the data were often incomplete or missing 
altogether. 



Research In 1980, the National Institute of Education developed an on-line system 

for monitoring its activities. This system lists basic data, including the 
time for the project, its funding, the contractor, and its topics. Since 
1980, the National Council on Educational Research has made the infor- 
mation from this on-line system available in its annual report. However, 
until 1985, the list did not include the projects carried out under the 
grants for the regional laboratories and national centers, preventing the 
detailed public examination of this important aspect of the work of nie. 
A special request was necessary to obtain this information. 

For information on activities prior to 1980, nie put together a historical 
data set that attempts to list the same items as given for its post-1980 
activities. However, the data set is not complete and cannot be relied on 
as the sole source of activities prior to 1980. Publications list the depart- 
ment's research activities in the 1970's, but these were not prepared for 
every year and are not easily available to other agencies or the genera) 
public. 



Statistics The National Center for Education Statistics could not provide us with a 

comparable list of information activities. Irregularly, it has published 
descriptions of its statistical program and plans. These are not available 
for 1984 to 1986, but in earlier years descriptions were provided in a 
companion volume to The Condition of Education , nces does, however, 
publish data from its statistical data series in its annual reports. The 
Digest of Education Statistics , Projections of Education Statistics , and 
Tne Condition of Education , but only selected portions of the data col- 
lected are reported. It may use only data from a few of the series and 
may mix these with data from other organizations. In addition, changes 
in the data sets cannot be determined from these documents. We relied 
upon selected publications and nces staff for this kind of information. 



Contract information can be tracked with the Annual Evaluation 
Report , mandated by the Congress (sections 417(a) and (b) of the Gen- 
eral Education Provisions Act of 1968). What is included in the report 
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has changed, however, so it was not possible to obtain comparable data 
across the 1970's to the present. It was not until 1975 that the report 
provided a record of current and recently completed activities that 
included titles, funding history by fiscal year, contract numbers, project 
monitors in the department, information on contractors, and starting 
and ending dates. In 1982, the department decided not to publish this 
comprehensive record of current and recent projects in the Annual Eval- 
uation Report , stating that a streamlined report including only current 
projects would be more useful to the Congress and the department. 
Because this more comprehensive report has been discontinued and 
because there was no computerized information management system as 
in NiE, we relied on department staff and project files for data on project 
history. 

Thus, information on the awards that have been made and funds that 
have been allocated is not readily accessible, even to those who spend a 
great deal of time and energy to obtain it. For others without these 
resources but with an interest in the information that has been produced 
and the way education information funds have been spent, a compre- 
hensive picture is nearly impossible to obtain. Our findings are based on 
the best available but nonetheless limited evidence. The analyses focus 
on the years for which our data are the most complete. 



Assessing Quality The longitudinal case examples serve two purposes. First, we summarize 

the current status of three types of statistical information with respect 
to their quality and changes in the level of quality over time. Second, we 
explore factors that may have contributed to variation in quality and to 
change. We also draw on additional examples pertaining to evaluation 
research and education to probe whether these findings can be general- 
ized to other types of information-gathering. 



Definition of Quality Our assessment focuses on four dimensions of quality: ( 1 ) relevance (Is 

the information useful for answering the types of questions posed by 
educational policymakers or other users?), (2) timeliness (b the infor- 
mation collected and reported when it is needed?), (3) technical ade- 
quacy (Is the information credible, free of egregious errors, and 
adequately reported?), and (4) impact (Has the information infiuenced 
decisions or actions relevant to educational processes?). Since specific 
definitions of these dimensions depend on the expected uses of the data, 
the nature of the information that is produced, and the data-gathering 
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mechanisms that are employed, we describe each one in detail for each 
individual case. 



Overview of the Case Examples We examined aspects of quality associated with three mechanisms for 

obtaining statistical information on the condition of education. Our first 
case is on the National Assessment of Educational Progress, naep began 
as a privately fimded survey and was later manda!:ed by the Congress. 
The second case represents a traditional statistical system in which a 
census is routinely taken and reported. The specific illustration of this 
type of data system was first implemented as the Elementary and Sec- 
ondary Education General Information Survey (elsegis) and has since 
undergone several changes in name and scope. It is now referred to as 
the Common Core of Data (CCD). Unlike naep, this system does not have 
a particular congressional mandate; rather, authority for this activity is 
derived from the general mission of nces. Our third case is on the Fast 
Response Survey System (frss). frss was developed in 1974 in an effort 
to provide timely information on specific policy issues. Unlike CCD and 
NAEP, it represents a series of unique surveys that vary in policy content 
and population coverage. Table 1.2 summarizes the msuor differences 
between these three data-gathering activities. These particular examples 
differ according to the origin of the activity, the collection method, peri- 
odicity (the frequency with which the data are collected), the basis of 
funding, and the topic area. 



In choosing these particular cases, we considered two factors. First, each 
case represents an established method for gathering information. There- 





Tabte 1 .2: QMeral AttributM of the ThrM Case Illustrations 


Case 


Collection method 


Authority 


Frequency of 
data collection 


Funding 


Topic area 


National Assessment c* 
Educational Progress 


National survey 


Congressional 
mandate 


Periodic with 
varying content 


Grant 


Achievement, student 
characteristics and 
attitudes, and educational 
environment 


Common Core of Data 


Census 


Agency mission 


Annual 


Cost sharing 


Local school districts and 
state agency 
characteristics, student, 
staff, and resources 


Fast Response Survey 
System 


Smalt-scale 
surveys 


Discretionary 
funds 


Penodic. based 
on demand 


Contract 


Varies 
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fore, it is possible to examine changes in quality over time and to 
identify factors that plausibly influence increases or decreases in 
quality. Second, the methods or procedures we examined represent 
three of the most common ways in which statistical information has 
been collected. 



Study LimitEtionS and limitations to our report should be noted. First, the findings are 

Qf ck rtf Vi based only on the information that was accessible in agency documents, 

OtrCngtiLS which were limited. 

Second, the case examples do not represent all education information; 
they exclude, for example, information from evaluation and research. 
However, they include m^gor initiatives that used or are using a good 
deal of the department's resources. In addition, they specifically repre- 
sent particular kinds of data collection strategies and serve as examples 
of the kind of work that can be accomplished, given a particular set of 
conditions (see table 1.2). 

Third, given limited time and resources, we could not do our own full 
assessment of the technical quality of the data. Instead, we drew on the 
writings of reviewers of education information activities and on the help 
of outside experts. We supplemented these with our own judgments on 
such matters as sample selection and analytic techniques. 

Our report has strengths as well. First, we looked at three kinds of infor- 
mation — research, statistics, and evaluation — m three different organi- 
zations — NiE, NCES, and opbe. It is rare that such an approach is taken; 
the focus is more tjrpically on one kind of information (such as statis- 
tics) or one type of information activity (such as a particular data set). 
Our approach allowed us a more complete look at the status of federally 
sponsored education information. 

Second, our longitudinal approach allowed us to look at changes in infor- 
mation activities over time and to track factors associated with periods 
of high and low activity. 

Third, while case studies did not give us a representative sample of 
information activities, they did allow us to look in-depth at sorre impor- 
tant statistical series. We were able to look at the origins of these series, 
the request strategy, implementation factors associated with particular 
negative or positive outcomes, the different co^ ;onents of quality, and 
critiques of the work. 
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Determining what information is produced and how it has changed over 
time is difficult. The data we found show, however, that federally spon- 
sored research and statistical and evaluative information on education 
has been declining during the past decade. Research and evaluation 
activities were hardest hit in terms of reductions in awards between 
1980 and 1985, showing overall declines of 65 and 79 percent, respec- 
tively. For statistics, the number of planned and ongoing surveys in 
three of four education areas declined by 31 to 42 percent between 1980 
and 1983. From 1974 to 1983, the interval between statistical data col- 
lections also changed, the time between collections increasing during this 
period. 

W<» also found changes in priorities. For research, there wr^ i shift away 
fium new data production to other activities such as dissemination of 
results and the provision of expert witnesses in civil r ^ts cases. For 
statistics, the shift was to the maintenance of core surveys. For evalua- 
tion, the shift was to smaller management studies. Many areas are no 
longer being studied, at least with respect to federal support for new 
inquiry, and they have not been replaced by other areas. New data col- 
lection appears to be particularly out of step with areas education 
experts identify as being in need of educational reform. Finally, there 
has been a shift in who is producing information, as shown, for example, 
by the shift from support for individual researchers to support for labo- 
ratories and centers in nie and the trend away from grants to contracts. 



A W5I tHq f nr found notable decreases in the number of awards across all three 

AWdTUb lOr units, nces reduced many of its activities. In nie and opbe, awards for 

Information Activities activities stood at less than one third and one quarter of their 1980 

levels, respectively. 



Research We examined the complete set of research activities in nie from 1980 to 

1985 only — all years for which adequate data were available. The activ- 
ities are those listed in ncer's annual reports. In this discussion, we sep- 
arate awards made to the mandated regional laboratories and national 
centers from all other grants and contracts (including those made in the 
three program areas, in the unsolicited proposal program, and in other 
units such as the office of the director). 

The number of awards from 1980 to 1985, as shown in table 2.1, 
declined from 476 in 1980 to 122 in 1984; they increased to 168 in 1985. 
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Table 2.1: Number of National Insmute of 
Education Awards for Fiscal Years 1980- 
85 by Program Funding Area 



Overall, from 1980 to 1985, the number of awards for information activ- 
ities declined 65 percent. Except for ''other'' awards (primarily those 
from the office of the director and the regional laboratories and national 
centers), all the program areas showed a marked decline in the number 
of awards from 1980 to 1985. The change from 1980 to 1985 was the 
most dramatic for the unsolicited proposal program because it was cut 
completely. The 84-percent decrease for educational policy and organi- 
zation was nearly as deep. Teaching and learning and the dissemination 
and improvement of practice declined 54 and 70 percent, respectively. 



Araa 


1980 


1981 


1982 


1983 


1984 


1985 


Teaching and learning" 


185 


118 


38 


49 


33 


85 


Educational policy and organization 


93 


61 


24 


37 


22 


15 


Dissemination and improvement of 
practice 


107 


74 


59 


49 


37 


32 


Unsolicited proposals 


58 


48 


35 


22 


0 


0 


Educational laboratories and national 
centers 


26 


26 


25 


27 


23 


18 


Other" 


7 


3 


4 


8 


7 


18 


Total 


476 


330 


185 


192 


122 


168 



^Includes centers that were not part of the regional laboratory and center network These were the 
centers on reaarng, teaching, and second language learning 

^'Covers miscellaneous awards not identified under any of the program funding areas listed, such as 
interagency agreements, awards made by the office of the director, and an award made in 1983 for the 
National Council on Educational Research 

Source National Council on Educational Research annual reports for fiscal years 1980 to 1985 



Statistics To obtain an overview of basic statistical data-gathering within noes, we 

relied on various reports describing nces programs and plans. ^ In this 
discussion, we focus on the portfolio of information-gathering activities 
in the four fundamental education domains — elementary and secondary 
education, higher education, adult and vocational education, and library 
resources— and specialized surveys (the National Longitudinal Survey 
and Fast Response Survey System). We consider other related activities 
in support of these efforts (such as quality control) separately. 

As seen in table 2.2, the total number of surveys nces planned and con- 
ducted grew by 49 percent (from 37 surveys to 55) between 1974 and 



^National Center for Education StatisUcs, The Condition of Education . Part 2, Programs and Plans 
(Washington, D.C: U.S Government Printing Office, 1980^); Projects, Products, and Services 
(Washington, D C: U.S. (5ovemment Printmg Office, 1974 and 1 976); and Fast ftesponse Safvey Sys- 
tem Reports , numbers 1 M7 (Washmgton, D.C.: VS. Government Printing (!>^lice, 
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1980. New surveys in adult and vocational education and in library 
resources accounted for most of this increase. The amount of data col- 
lection (as measured by the number of ongoing and planned surveys) in 
elementary and secondary education and higher education remained 
roughly constant. Between 1980 and 1983, the last year for which data 
permitting this analysis were available, data collection activity declined 
by 31 percent, returning to its 1974 aggregate level. In three of the four 
major education areas, the number of planned and ongoing surveys 
declined by 31 to 42 percent in this same period. 



Table 2.2: Number of National Center for 
Educational Statietics Surveys for 
Selected Years 1974-83* 



Area^* 


1974 


1976 


1980 


1981 


19b2 


1983 


Primary, elementary, and secondary 
education 


12 


18 


12 


9 


9 


7 


Postsecondary education 


9 


10 


8 


7 


P 


8 


Adult and vocational education 


9 


11 


14 


15 


12 


9 


Library resources 


6 


13 


16 


13 


10 


11 


National Longitudinal Survey 


1 


1 


2 


2 


2 


2 


Fast Response Survey System 


0 


2 


3 


1 


3 


1 


Total 


37 


55 


55 


47 


44 


38 



^Excludes the NatK)nal Assessment of Educational Progress (NAEP was an activity of NCES in 1974 
and 1976 and was transferred to NIE in 1978, no other NCES rebponsibility changed this way ) 

^ime did not permit our verifying information the department provided in response tc a draft of this 
report Therefore, we have not included the analyses in our tables or discussion The Department of 
Education reported data for 1984-85, as follows primary, elementary, and secondary, 10 (1984), 8 
(1985). postsecondary, 8 (1984), 6 (1985), adult and vocational, 0 (1984 and 1985); library, 1 (1984 and 
1985), Fast Response Survey System, 3 (1984 and 1985) 

Source Department of Education data for 1984 and 1985. National Center for Education Statistics, 
Projects, Products, and Services (Washington, DC US Government Printing Office. 1974 and 1976), 
The Condition of Education , part 2, Programs and Plans (Washington, DC US Government Printing 
Office, 1980-83). and ^^sffiesponse Survey System Reports , numbers 1-17 (Washington, DC US 
Government Printing Office, 1980-84). 



We also examined the frequency of data collection — that is, periodicity, 
or the inter/als between time points when data are collected. As table 
2.3 shows, from 1976 to 1980, scheduled periodic surveys increased and 
both annual and occasional one-time surveys decreased. Between 1980 
and 1983, the percentage of periodic surveys declined to the 1974 levels. 
The result of these changes over 9 years was a 20-percent reduction of 
annual surveys (which permit: detailed analysis of trends), an 83-per- 
cent increase in occasional one-time surveys, and little change m peri- 
odic surveys. 
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Table 2.3: Schedule for the National 



Center for Education Statistics Data Percentage of surveys 

Collection for Selected Years 1974-83- y^ar surveys Annual Periodic Occasional one tim e 

1974^ 37 41% 41% 16% 

1976 55 31 35 35 

1980 55 27 49 24 

1981 47 36 40 23 

1982 44 30 43 27 

1983 ~38 32 39 29 



^Tirne did not permit our verifying information the department provided in a response to a draft of the 
report Therefore, we have not included the analyses in our tables or discussion The Department of 
Education reported data for 1984-85, as follows number of surveys, 26 (1984), 20 (1985), percent 
annual, 42 (1984), 55 (1985), percent periodic, 35 (1984), 25 (1985), percent occasional one time. 23 
(1984). 20(1985) 

^Periodicity could not be determineo for one survey in 1974 

Source Department of Education data for 1984 and 1985, National Center for Education Statistics. 
Projects, Products, and Services (Washington, D C U S Government Printing Office. 1974 and 1976), 
The Condition of Education , part 2, Programs and Rans (Washington, D C U S Government Printing 
Office. 1980-83). and Fast"Responsr5urvey System Reports , numbers 1-17 (Washington. DC US 
Government Printing Office, 1980-84) 



Evaluation Contract activities for the Office of Planning, Budget, and Evaluation 

from 1975 to 1985 are presented in table 2.4. The activities shown here 
were ongoing or received funding during the fiscal year. The high level 
of activity that began late in the 1970's (80 or more awards annually) 
peaked in 1980 and began to drop in 1981, leveling off to 25-28 activi- 
ties annually. Overall, the decline from 1975 to 1985 was 73 percent 
From 1980 to 1985, the decline was 79 percent. 
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TaM 2.4: Number of Office of Planning, Budget, and Evaluation Awards for Fiscal Years 1975-85 by Program Funding Area* 


Area 


1975 


1976 


1977 


1978 


1979 


1980 


1981 


1982 


1983 


1984 


1985 


Elementary and secondary education 


28 


53 


61 


53 


56 


75 


47 


39 


12 


13 


13 


Postsecondary 


26 


21 


26 


33 


11 


19 


12 


8 


8 


4 


3 


Occupational, handicapped, and 
developmental 


8 


5 


5 


23 


14 


18 


13 


6 


1 


4 


4 


Program assessments 


• 


• 


• 


• 


• 


4 


4 


2 


• 


• 


• 


Adult and vocational education 


11 


8 


8 


• 


• 


• 


• 


• 


• 


• 


1 


Libraiies and education technology 


2 


3 


5 


• 


• 


• 


• 


• 


• 


• 


• 


Miscellaneous 


18 


16 


5 


4 


3 


3 


1 


2 


7 


6 


4 


Total 


93 


106 


110 


113 


84 


119 


77 


57 


28 


27 


25 



^Awards were ongoing or had funds obligated in the year oi the report, empty celts denote the absence 
of data in a particular category for a particular year From 1983 to 1985, awards were listed differently 
than in previous years These awards were recorded for crossyear comparisons based on information 
provided in the annual evaluation reports After 1977, awards in the occupational, handicapped, devel- 
opmental, vocational, and adutt education categories were combined 



Source U S Department of Education. Office of Planning, Budget, and Evaluation. Annual Evaluation 
Report (VWashington, D C U S Government Printing Office, 1975-85) 

Because of a change in reporting during our period of study, we had to 
reclassify studies appearing in 1983 through 1985 according to the ear- 
her program designations. With this reclassification, we could look 
across areas from 1975. Within elementary and secondary education 
and postsecondary education programs, we see declines of 54 and 88 
percent, respectively, in the awards from 1975 to 1985. The biggest 
overall decrease followed the passage of block grant legislation, which 
affected many of the elementary and secondary education programs 
that had previously received the bulk of the evaluation support and 
review.^ 



When considered in total, the production of information has declined 
dramatically during the past decade. We did not find a substitution of 
one kind of information for another. Rather, by 1985 there were fewer 
research, statistical, and evaluation activities. 



^In 1981, the Congress passed the Omnibus Budget Reconciliation Act (Public Law 97-35), consolidat- 
uig 80 federal categorical programs into nine block grants to the states: 38 categorical grants for 
education were converted under chapter 2 of the Education Consolidation and Improvement Act of 
1981. This act eliminated other reporting requirements and evaluation activities, and legislative set- 
asides for evaluations of five of the programs that were folded into the chapter 2 block grant were 
rescinded. 
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In addition to the general transition in recent years to fewer awards for 
information, the data on research, statistics, and evaluation suggest that 
the mqjor informr^^on-producers have made changes in priorities in 
recent years, shifting away from the production of new data, and in the 
areas of study. 



Shift Away From New 
Data Production 



Research The National Institute of Education funded a variety of activities every 

year. For example, it funded dissemination activities (such as those of 
the 16 Educational Resources Information Centers (eric) clearing- 
houses), demonstration projects, commissioned papers, syntheses of 
prior research and evaluations, and expert panels to help review appli- 
cations for awards. In our effort to describe the character of and 
changes in the department's research function, we looked at the types of 
activities nie funded. First, we compared 1980 awards to 1985 awards, 
excluding laboratory and center awards (which we discuss separately in 
the next section). Our results appear in table 2.5. A random sample of 
these awards is listed in tables I.l and 1.2 in appendix I. 



Table 2.5: National Institute of Education 
Awards for 1980 and 1965 by Type of 
Activity 





Percentage of awards 


Activity* 


1980 


1985 


New data collection 


65% 


11% 


Dissemination 


22 


43 


Other" 


13 


46 


Totai nuinlMr of awards 


450 


157 



^Excludes the operation of the mandated laboratories and centers 

^Includes planning, development, conferences, and support for experts for proposal reviews 
Source National Council on Educational Research annual reports for fiscal years 1980 and 1985 and 
data from the Office of Educational Research and Improvement 



Looking at the distribution of awards across these years, we found a 
very large shift away from new data collection to other activities. In 
1980, 293, or 65 percent, of the total 450 nonlaboratory and noncenter 
awards were for new data collect: By 1985, this number had shrunk 
to 17 of 157 awards, or 1 1 percent of the total. 

In absolute terms, activities other than new data collection, such as dis- 
semination, declined. Dissemination awards, for example, dropped from 
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98 in 1980 to 68 in 1985. In relative terms, however, these other activi- 
ties received the m^or share of the institute's nonlaboratory and 
noncenter awards in 1985. Dissemination increased notably, from 22 
percent of all such awards in 1980 to 43 percent in 1985. 

"Other" activities also increased notably from 13 percent in 1980 to 46 
percent in 1985. The types of **other" activities funded in both years 
included research syntheses, commissioned papers, study groups, and 
expert panels. Activities that were awarded funds in 1985 but not in 
1980 included the development of technology, computer software, and 
curricula and the use of expert witnesses to give legal testimony and 
write depositions for civil rights cases. Types of activities funded in 
1980 but not in 1985 included the development of data files, archives, 
and fellowships. (In commenting on this report, the department claimed 
that the items under **Other" in table 2.5 (computer software, witnesses 
in civil rights cases, and so on) were inaccurate, but we coded them 
directly from oeri's information system and have not deleted them, as 
the department suggests.) 



Laboratories and CJenters We examined activities involving new information collection versus dis- 

Activities semination-related and other work for the regional laboratories and 

national centers. In the entries for the nie on-line computer system, labo- 
ratories and centers report the percentage of the costs of specific 
projects devoted to basic and applied research, dissemination, and other 
activities. Thus, we could analyze the resources placed in each area 
across projects. Our results are shown in table 2.6. The laboratories 
have historically been charged with conducting regionally relevant work 
and with translating research into practice. The centers, although 
charged with the creation of new knowledge, have also been responsible, 
historically, for some dissemination and efforts to improve practice. 

As table 2.6 shows, the laboratories were putting proportionately less 
money into new data collection in 1985 than in 1980, a decline from 34 
percent to 24 percent of the total awards. The proportion of resources 
allocated to dissemination increased sharply, from 29 percent in 1980 to 
41 percent in 1985. The centers continued to allocate more than half 
their resources to new data collection; however, dissemination 
increased, from 12 percent in 1980 to 21 percent in 1985. 

The aggregate figures mask substantial variability in the activities of 
the laboratories and centers projects. Looking at the activities associated 
with projects within each laboratory, we find that 67 percent of all labo- 
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Tabto 2.6: Percentage of Obligations HBSB^B^^^H^^^^^^^^^^^^H^^^B^B^H^^^^H 

AMignad to l^boratoriea and Canters Percentage of awards 



Activity* 


1980 


1985 


Laboratories 


New data collection 


34% 


24% 


Dissemination 


29 


41 


Other 


37 


35 


Total» 


$50 


$39 


Centers^ 


New data collection 


62% 


56% 


Dissemination 


12 


21 


Other 


27 


22 


Total» 


$25 


$28 



^Excludes 1985 data for the Educational Technology Center and the North Central Regional Education 
Laboratory 

^uttiyear awards in millions of current dollars 

^Excludes data from centers supported by NIE's discretionary funds 

Source Department of Education Office of Research Computer Management Information System 



ratory-sponsored projects in 1980 involved some funds for dissemina- 
tion, but by 1981s 96 percent of all projects devoted funds to this activ- 
ity. Furthermore, across the projects, the median allocation of funds 
spent on dissemination rose from 20 percent to 30 percent between 1980 
and 1985. 

The shift away from new data collection by the research units may have 
serious long-tenn consequences for education information. At present, it 
is possible to use prior research to address questions when new data 
collection has not been undertaken. However, prior research may 
quickly lose its relevance or it may be too low in technical adequacy to 
sustain continued reapplication to new questions. New data must con- 
stantly be produced to meet both departmental and congressional infor- 
mation requirements and to provide up-to-date information to 
disseminate to teachers and other practitioners. If it is not, information 
will be forgone and policies will be based on less than the most complete, 
relevant, and timel: lata. 



29 



ERIC 



PMe 27 GAO/PEMIV8M The Prodactioii and Quality of Edncatkm Intamatlmi 



Chaptei 2 

Hie Prndnctioii of Information 



Statistics Until recently, new statistical data collection efforts in the department 

have received less priority than other activities, including the mainte- 
nance of existing data systems. Data for this discussion come from 
recent reviews we and the Congressional Research Service have made of 
the federal statistical system between 1980 and 1984.^ These studies 
reported that nces reduced its information activities over this period. In 
our 1984 study, we found that two criteria were used in the decisions to 
reduce data collection: whether or not an activity was part of a core 
program and whether or not the data collection had a congressional 
mandate or yi^s a departmental requirement. 

The core surveys (those that provide the basic information on student 
staff, and institutional characteristics or carry out the nces mission) 
were given highest priority. The Common Core of Data and the higher- 
education information system were included as core surveys. Areas that 
underwent reductions included technical assistance, library services, 
and statistical research. New initiatives in data coUection and efforts to 
improve statistical methodology were also significantly reduced. 

Specific efforts that either scaled back or eliminated ongoing data coUec- 
tion activities are reported in table 2.7. Some of the reductions, such as 
the delay in the noncoUegiate postsecondary school survey, led to gaps 
in education data. Decreases in sample sizes and the frequency of data 
collection call into question the precision of the resulting data. Validity 
studies, which had previously been made on some surveys, were also 
eliminated. (Specific changes in nces primary, elementary, and second- 
ary school surveys are reported in table II. 1 in appendix II.) 

In reporting changes in nces surveys, we do not imply that these are 
necessarily problems. Some series may no longer be valuable; new series 
may be needed; continued surveys may be organized and sequenced in 
ways that are less burdensome and permit more useful analyses across 
surveys. However, the changes between 1980 and 1984 were not part of 
an external, systematic review of statistical needs. 



^U.S. House of Representatives, Committee on Government C perations, The Federal Stetistical Sys- 
tem 1&80 to 1986 (Washington, D.Q: November 1984), and U.S. General Accounting Office. Status of 
the Statistical Community After Sustaining Budget Reductions. GAO/IMTEC-84-17 (Washington, 
D.C July 18. 1984). 
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Tabto 2.7: Changes in Education Survaya 
In Fiscal YMra 1981-84 



Action 


Survey 


Savinga 


Eiimination 


Technical assistance grants to states 


$350,000 




Survey of recent college graduates 


224.000 




1982 teacher dennand-and-shortage survey 


175.000 




National Vocationat Education Data System 


a 


Reduction 


Precision 


National longitudinal survey of high school and 
beyond 


$225,000 


Coverage 


Parts of the Higher Education General 
Information Survey 


a 


Periodicity 


Student residence and migration survey 


a 




Library general information survey 


a 


Substitution 


Supplement to current population survey 
replaced data on students in noncollegiate 
postsecondary schools 


a 




Nonfederal support replaced NOES support for 
national longitudinal survey of the high 
school class of 1972. fifth follow-up 


a 


Delay 


Analysis of private school survey 


$200,000 




Noncollegiate postsecondary school survey 


225.000 



^Not available 

Source Department of Education. U S General Accounting Office. Status of the Statistical Community 
After Sustaining Budget Reductions . GAO/IMTEC-84-17 (Washington. 0 C July lfl. 1M4). pp 52-54 



In late 1985, a commendable effort was made to undertake such a sys- 
tematic external review and CES initiated an internal redesign of its data 
collection efforts. In December 1986, CES began implementing one prod- 
uct of these reviews — the Elementary and Secondarj . .brmation Data 
System. As it is now planned, this system will incorporate current 
surveys and six new surveys into one system (see appendix II, table 
II.2). 



Evaluation Like the department's research function, the Office of Planning. Budget, 

and Evaluation is involved in a variety of activities that include new 
data collection and technical assistance to the states. We compared the 
types of activities for 1980 and 1985 and identified changes in the dis- 
tribution of activities. Table 2.8 and figure 2.1 present the results of our 
analysis. (Random samples of contract awards for 1980 and 1985 are 
listed in tables 1.3 and 1.4.) 




The number of evaluation activities producing new data has declined 
substantially. In 1980, there were 59 new data activities. In 1985, there 
were 18 — a decline of 69 percent. In tenns of total contract activities, 
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Tabl« 2.8: Number of Offico of Planning, 



Budgot, and Evaluation Contract Awards Parcantaoa of awarda 

for Racal Yaara 1980 and 1985 by Type Activity 1M0 1985 

of Activity New data collectton 50% 72% 

Technical assistance to states for data collection and 
refinement in data collection systems 29 12 

Other* 21 16 

Total number of contractt 119 25 

^Includes planning, data file development, and research support 

Source Department of Education, Office of Planning, Budget, and Evaluation, annual evaluation reports 
for fiscal years 1980 and 1985 



Figure 2.1: Office of Planning, Budget, and Evaluation Compensatory Education Obligations for Fiscal Years 1972-84 by Type of 
Activity 



9 DoHart in Millions 
6 

7 




in 




1972 1973 1974 1975 1976 1977 197S 1979 19S0 1991 19S2 1993 1984 



I New data collection 

State reftnements to evaluations 
Models for evaluation 
Technical assistance centers 



Source: Office of Planning. Budget, and Evaluation annual evaluation reports for 1972 and 1984 and 
data provided by OPBE staff 
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however, the percentage increased. New data collection represented 
50 percent of total contract awards in 1980; in 1985, it represented 
72 percent of the total. As can be seen in table 2.8, opbe funded very 
few other activities in 1985. 

In at least one area — compensatory education for the disadvantaged — 
OPBE turned almost completely away from its involvement in evaluation. 
Tracking the activities over time, we found that a greater proportion of 
its compensatory education resources were being taken up by technical 
assistance work — that is, technical assistance centers, models develop- 
ment, work related to the Title I Evaluation and Reporting System 
(TIERS), and state refinemaits to Title I evaluations (see figure 2.1). By 
1984, only about 9 percent of the total funds spent on compensatory 
education (about $180,000 in constant dollars), supported anything 
other than technical assistance. 



Shift in Focus The concern that new data will not be available for future departmental 

information needs led us to compare the 1980 and 1985 new data collec- 
tion awards by area of study. We were interested in the areas that were 
no longer targets for new information collection and in identifying areas 
that have been most recently emphasized. 



Research Table 2.9 shows the new data research awards coded by area for all 

discretionary awards. Every area saw a substantial reduction in the 
number of awai^ds. In 1980, for example, 56 of the 293 awards for new 
data went toward studies of special populations such as minorities and 
women. In 1985, there were five such studies. Some areas such as learn- 
ing in the home, at school, in the community, and at work and what we 
identified as "school problems," including such issues as dropouts and 
delinquency, received no new data funds at all. The only area that was 
added for 1985 awards was education standards and only one study was 
done in this area. 




These reductions in new data collection are particularly a problem when 
looked at as areas that appear to be in most need of educational reform. 
In a report prepared for the National Council on Educational Research 
early in 1984, the Onter for Leadership Development (cld) outlined 
areas seen as priorities. From its review of eight major national reform 
studies, a sur^rey of 72 educational experts, regional meetings held at 
laboratories and centers iii 1983, and two departmental assessments, cld 
identified the most critical areas as follows: 
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TaM 2.9: Number of Natk>fMil Institiite of 
Education Indlvlduai Rasaarch Awarda 
for Now Data Coltaction in 1980 and 1985 
by Araa of Study 



Mraa* 


1980 


19SS 


Special populations 


Disadvantaged 


2 


3 


Bilingual 


13 


1 


Language minonty 


3 


1 


Handicapped 


7 


0 


Minority 


9 


0 


Blacks 


8 


0 


Native Amencans 


3 


0 


Females 


10 


0 


Hispanics 


1 


0 


Total 


56 


5 


Area studies 


Reading and writing 


40 


2 


Math and science 


23 


1 


Language acquisition 


5 


0 


Social studies 


1 


0 


Literacy 


7 


1 


Total 


76 


4 


Teaching and instruction 


Teachers 


14 


1 


Instruction 


7 


0 


Effective schools 


12 


3 


Total 


33 


4 


Learning in nonschool settings 


Parents and family 


6 


0 


Work 


2 


0 


Play and work 


1 


0 


School, home, and community 


3 


0 


Adult education 


4 


0 


Total 


16 


0 


Technology 


Computer technology 


2 


1 


Other education technology 


3 


0 


Radio and television 


1 


0 


Total 


6 


1 



(continued) 
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ATM* 


1980 


1985 


School finance and resources 


Finance 


3 


0 


Resources 


2 


0 


Total 


5 


0 


Education administration 


Leadership and management 


9 


0 


Education administration 


10 


0 


School district 


2 


0 


Elementary and secondary education 


1 


0 


Postsecondary education 


3 


0 


Total 


25 


0 


School problems 


Dropouts 


1 


0 


Delinquency 


2 


0 


Desegregation 


7 


0 


Equity 


4 


0 


Mainstreaming 


1 


0 


Declining enrollments 


2 


0 


Total 


17 


0 


Student outcomes 


Achievement 


2 


1 


Cognition 


8 


1 


Social learning 


1 


0 


Student success 


2 


0 


Careers 


3 


0 


Total 


16 


2 


Testing and assessment 


17 


0 


Research on dissemination 


Dissemination 


2 


0 


Educational Resources Information Centers 


2 


0 


Total 


4 


0 


Other 


Miscellaneous 


12 


0 


Policy 


10 


0 


Education standards 


0 


1 


Total 


22 


1 


Total 


293 


17 



^his list includes nonlaboratory and norK:enter awards 
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**Improved teacher preparation is probably the most important, followed closely by 
a need to strengthen curriculum in mathematics, science and English. More effective 
instruction, classroom management and school leadership are frequently mentioned 
along with a continuing concern for a definition of the federal role in education."^ 

These were seen as the priority areas, but there were few awards for 
new data collection on these topics in 1 985. 

As we indicated earlier, the centers and, to some extent, the laboratories 
do collect new information. In 1985, they were almost the sole source for 
up-to-date knowledge. The topics of laboratory and ct^nters programs 
were reported in terms of nie priorities for 1980-85. However, ambigui- 
ties in the available data prevented detailed analyses of the 1980 and 
1985 awards for changes in research topics. Detailed information was 
not available that would have permitted us to determine how many 
awards were directed at various disadvantaged populations, for exam- 
ple. In 1985, competition led to awards for centers for topics that taken 
broadly, and together with nie ''discretionary** centers for reading, 
instruction, language learning, and technology, are more closely related 
to the priorities that might be derived frcHU the educational excellence 
and reform initiatives cld identifled. 



Elvaluation In looking at the areas of new evaluation data collection for opbe, we 

found the breadth of coverage diminished in 1985 compared to 1980. As 
table 2.10 shows, contract activities in 1985 covered special populations, 
higher education, elementary and secondary education, vocational edu- 
cation, and rehabilitation. In addition, awards were given for work on 
policy issues of interest to the department. In 1980, awards were given 
in most of these areas as well as many others, including teachers, school 
fmance, desegregation, basic skills, and library and education resources. 
For areas in which opbe awarded contracts in 1980 and 1986, the pro- 
portion of awards remained roughly the same. For example, 41 percent 
of its contracts were awarded for work with special populations in 1980, 
and 44 percent went to the same area in 1985. 



^Center for Leadership Development, Creating and Disaeminating Knowledge for Educational Re form: 
Prticy Manage*nent of the National Instituie of KQucation s Kegkmal Kducationai LaDoratorics a nT 
]>iationai Kesearch and Development uentera (Los Angeles, (Jaii:.: retmiary i W4), p. IT 
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Tabto 2.10: Mumb^r of OfflM of Planning, 
Budgot, and Evaluation Naw Data 
Contract Awarda for 1980 and IMS by 
Aim of Study 



Women 


1 


0 


Neglected and delinquent 


t 


0 


Migrant 


1 


0 


Total 


24 


8 


Area studies: basic skills 


3 


0 


Teaching and instruction 


Teachers 


1 


0 


School improvement 


1 


0 


Total 


2 


0 


Learning in nonschool settings 


Parents 


2 


0 


Adult and career education 


2 


0 


Community education 


1 


D 


Total 


5 


0 


School resources and finance 


2 


0 


Education administration school districts 


1 


0 


School problems: desegregation 


4 


0 


Policy- department issues 


0 


3 


Postsecondary education 


3 


1 


Student loans and grants 


5 


0 


College enrollment 


0 


1 


Cooperative education 


1 


0 


Total 


9 


2 


Elementary and secondary 


0 


1 


Chapter 2 


0 


1 


Total 


1 


1 


Vocational education 


1 


1 


Rehabilitation 


0 


3 


Dissemination. National Diffusion Network 


1 


0 


Training 


2 


0 


Federal educatk>n programs 


1 


0 


Library or education resources 


2 


0 


Program evaluatk>n 


1 


0 


Total 


59 


18 



ATM 


1980 


1985 


Special populations 


Disadvantaged 


12 


0 


Bilingual 


4 


3 


Language minority 


1 


2 


Handicapped 


2 


1 



Native American 
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In addition to the analysis of contract awards, we had additional data 
with which to describe the office's activities. For 1980 and 1984, we 
surveyed all federal nondefense evaluation units on their program eval- 
uation activities.^ opbe responded to both surveys, allowing us to com- 
pare the nature and scope of its activities in these 2 years. 

In 1980, 15 percent of staff time was devoted to internal evaluations, 
and 60 percent was consumed by planning and monitoring external eval- 
uations. In 1984, no staff time was devoted to internal evaluations, 
external evaluations accounted for 40 percent of staff activities, and 
policy analysis consumed 35 percent of staff time, a new activity. Policy 
studies were not conducted in 1980. 

Other changes included a decrease in new data collection and a shift to 
smaller-scale studies. The respondent to the 1984 questionnaire indi- 
cated greater reliance on the use of existing data sources, including data 
from other federal agencies and departments, private companies or cor- 
porations, and private interest groups and associations. 

The 1984 survey asked respondents to assess the extent to which the 
evaluation function within their units had changed since 1980 and to 
interpret the changes. The respondent for opbe reported a shift to 
smaller-scale management studies (for example, policy analyses). Sev- 
eral reasons for this shift were noted: (1) the conversion of categorical 
grants to block grants, (2) opbe's no longer having enough funds to con- 
duct discretionary evaluation activities, (3) the belief that management 
and policy-oriented studies were as useful, and (4) more specific and 
focused departmental requests for this type of information. The shift 
was not attributed to a lack of staff or to no need for program evalua- 
tion (under the idea that programs had already been sufficiently evalu- 
ated or were terminated). 



Inf Onnd.tiOn Producers other notable shifts in informatlon-related activities were a shift in 

research production away from individual researchers and agencies 
toward the laboratories and centers and a shift from grants for 
researchers toward contracts. 



A General Accounting Office, Federal Evaluations: Fewer Units, Reduced Resources, Different 
Studies From 1980 , GA0/PEMIMJ7-» (Washington, D C. January 2S, ld87.) 
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Research For research, we compared laboratory and center awards (that is, 

awards to department-sponsored institutions) in the three program 
areas — teaching and learning, educational policy and organization, and 
dissemination and improvement of practice — with awards to individual 
researchers and public and private agencies for 1980 and 1985 (see 
table 2.1 1). Overall, the proportion of awards made to department-spon- 
sored institutions increased substantially from 1980 to 1985. The insti- 
tutions carried out 25 percent of the awards in the three program areas 
in 1980, compared with 56 percent in 1985. Thus, nie increasingly relied 
on the laboratories and centers for information production. 



Table 2.11: National Inatitute of 
Education Awarda to Inatitutiona and 
individuals in 1980 and 1984 



Recipient 

Institutions^ 



Percentage of awaida 
1980 1985 



25% 



56% 



individuals^ 



75 



44 



Total number of awarda^ 



492 



282 



^Regional laboratories, national research centers, discretionary research centers, and education infor- 
mation centers 

^'Individual investigators m universities, research organizations, and similar agencies such as nonprofit 
educational advocacy groups and state education organizations 

^Includes teaching and learning, educational policy and organization, and disserr.nation and improve- 
ment of practice 

Source National Council on Educational Research annual reports for 1980 and 1985 and the Office of 
Research on-line system 



Using NIE data, we examined the proportion of work funded through 
contracts and grants, excluding awards for laboratory and center opera- 
tions. We compared 1980, the earliest data we had available, to 1985, 
the most recent data. The use of contracts increased proportionally and 
substantially. In 1980, only 23 percent of the awards were crntracts; in 
1985, this rose to 86 percent. For grants, both the actual number and 
proportion of total awards declined precipitously: 336 awards fell to 17 
awards and 75 percent fell to 1 1 percent. 



Statistics nces funded its work through contracts rather than grants, nces did not 

fund its work through institutions such as special centers in either 1980 
or 1985, although it did support tong-term projects such as the State 
Education Assessment Center, operated by the Council of Chief State 
School Officers. 
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Evaluation The Office of Planning, Budget, and Evaluation made awards in both 

1980 and 1985 primarily through contracts rather than grants. How- 
ever, in 1980, the award mechanisms were more diverse. In recent years, 
OPBE has made long-term awards for policy analyses. These were not, 
however, comparable to awards to laboratories and centers, since the 
OPBE awards are used for the studies specified by the department that 
provide data collection and analytic support with a fast turnaround and, 
therefore, are more comparable to contracts. 



Suininary The cumulative result of the shifts in awards is that the m^uority of the 

department's information producers are institutions or contractors. That 
is, since ncss and the Offlce of Planning, Budget, and Evaluation never 
made many awards for grants to individual researchers, nie was the pri- 
mary source of such support. This source, in essence, dried up during 
the period of our review. 

In terms of the implications for educational information, contracts typi- 
cally involve a greater specification of the questions to be investigated 
and study design. Also, the products of contracts are typically reviewed 
by the funding agency before release, whereas the products of grants 
are typically required after release. While contracts may be most appli- 
cable when there is a specific request for information (for example, a 
congressionally mandated study) or when continuity in data gathering is 
necessary (for example, in a statistical series), their use as the predomi- 
nant vehicle for funding research is likely to constrain inquiry. 
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If information is to inform debates, guide actions, or assess changes, it 
has to be high in quality. We reviewed evidence regarding four dimen- 
sions of quality — relevance, timeliness, technical adequacy, and 
impact— for the National Assessment of Educational Progress, the Com- 
mon Core of Data for elementary and secondary education, and the Fast 
Response Survey System. We assessed changes in quality and factors 
associated with them in each program. In this chapter, we present our 
case studies on these three programs and then describe practices associ- 
ated with each dimension of quality. 

In general, naep ranked high on all four quality dimensions, but it has 
suffered some decline in relevance and timeliness in adapting to fiscal 
constraints, ocd was not rated high on any of the four indicators. Data 
were not comparable across states; mainly, they were reported at differ- 
ent levels of aggregation or used different definitions and procedures. 
Further, we could find little evidence on the use of ocd in policy deci- 
sions. Problems with CCD have long been recognized, but few have been 
solved. FRSS was rated moderate to high on quality, especially given the 
low budgets associated with each survey. It was responsive to the infor- 
mation needs of the requester and minimized time delays by releasing 
findings early. It appeared to be technically adequate, but the reporting 
of procedures could be improved. 

The case studies reveal several practices associated with high ratings on 
the quality dimensions. Relevance was increased through the addition of 
data elements, the tailoring of data collection to the information request, 
and flexible dissemination. Timeliness was improved by early release of 
data and diverse formats for dissemination. Technical adequacy was 
improved through appropriate quality-control procedures and the use of 
research to assess the credibility of the data. 



The National 
Assessment of 
Educational Progress 



Purpose and Background naep is a congressionally mandated survey of the knowledge, skill, 

imderstanding, and attitudes of young Americans. Although the survey 
was not mandated until 1978 (20 U.S.C. 1221e), the department began 
funding naep data collection in 1968. Since then, more than 1 million 9-, 
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13-, and 17-year-olds and adults 26 to 35 years old have been assessed. 
Assessments have been conducted in 10 m£uor school-related areas, but 
each content area has been assessed at staggered and varying intervals. 
Because of its sampling format, naep is flexible with regard to topic cov- 
erage and the target population that is surveyed. On several occasions, 
small-scale assessments have been added to the naep sampling frame 
and data collection procedures (for instance, the young-adult literacy 
assessment funded by nie). naep's topic coverage and schedule since 
1969 are in table III.2. (The funding history is in table III.l.) 

The purposes of naep have changed over time, naep was originally con- 
ceived of as a means of obtaining a national accounting of educational 
progress. Because fears were expressed that naep could be used to 
devise a national curriculum and thereby encroach on the states' author- 
ity, the founders of naep deliberately devised the assessment so that it 
could not be used to derive state-to-state comparisons. Also, the original 
assessment format could not provide an overall score for an individual 
student. (Because each student was not tested on all items, matrix sam- 
pling of items was used in constructing the test.) 

To minimize federal intervention, naep was originally conducted by a 
state-based consortium — the Education Commission of the States. 
Before 1979, federal funding was portioned out by cooperative agree- 
ments between nces and the conunission. In response to the 1978 con- 
gressional mandate, nie assumed responsibility for naep and initiated a 
competitive grant framework. The only bidder was awaided a 3-year 
grant. After a two-stage competition in 1983, the Educational Testing 
Service (ets) won a 5-year grant. 

naep was awarded about $6 million in 1985, similar to the allocation in 
1972. However, the current purchasing power is about $2.4 million, a 
59-percent decline. 



Relevance Over the past decade, naep's relevance to federal, state, and local stake- 

holders has been a main reason for criticism. Over the past several 
years, naep has tried to address this concern by collecting extensive data 
on students' backgrounds, attitude variables, and educational condi- 
tions; expanding its policy committee's role in the review and develop- 
ment of background and attitude questions; and increasing the 
dissemination of and technical assistance for NAEP-generated material to 
states and local school districts. 
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However, other changes in naep's design have made it less relevant for 
answering certain types of questions. In particular, in 1969-73, five tar- 
get populations (9-, 13-, and 17-year-olds in and out of school and 
adults) were assessed annually (see table III.3). In later years, from 1977 
on, the number of target populations was reduced from five principal 
groups to three (9-, 13-, and 17-year-olds who remained in school). 
Assessments for specialized groups (fw example, dropouts) were con- 
ducted on only two occasions in the past decade. In our 1976 assessment 
of NAEP, we attributed the decision to suspend data collection for young 
adults to budgetary restrictions.' At the time of that review, this action 
was characterized as temporary. The pattern of assessment since 1976 
suggests that budgetary restrictions have had a lasting effect. 

The relevance of naep for assessing change is inherently limited by the 
frequency of the data collection. Because the time intervals between 
assessments have been lengthened, naep's ability to examine specific 
types of questions has diminished. For example, in a recent report on 
educational achievement, the Congressional Budget Office asserted that 
although NAEP has been able to document long-term trends in achieve- 
ment, the intervals between assessments are too wide to ascertain pre- 
cisely when declines or increases occurred.^ The frequency with which 
an area can be assessed is also limited by the nature of the assessment 
process; if the interval is too brief, there may not be enough time to 
analyze and interpret the data. Further, capitalizing on lessons learned 
from each assessment to improve subsequent assessments might also be 
hindered with shorter testing intervals. 



Timeliness Timeliness can be thought of in two ways: the timeliness of the assess- 

ment and the timeliness of reporting and disseminating other informa- 
tion products such as technical reports, bulletins, and public-use data 
tapes. 



Timeliness of Assessment As we already noted, naep's skill areas have been assessed in rotation. 

This means that the most recent data available for reading may be more 
than 2 years old and for other ^; eas, such as career arid occupational 
development, up to 12 years old. Furthermore, for areas that have been 



*U.S. General AcoHinting Office, The National Assessment of Educational Prog ress; Its Results Need 
to Be Made More Useful ^ GA0/HKl>7t)-l 13 (Washington, DC: July ld76) 

^Congressional Budget Office, Trends in Educational Achievement rwashington, D.C: April 1986). 
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reassessed, the intervals have been variable. (The pattern of testing 
over the history of naep is given in table III.2.) 

Several features of the assessment schedule are worth noting. Reducing 
the 10 **content domains'' to the five "core" areas (reading, science, 
mathematics, social studies and citizenship, and humanities) was the 
result of budgetary constraints. Further, whereas prior to 1980 naep 
9ssessments were conducted annually, the interval between assessments 
increased from 1 to 2 years in 1980. Budgetary restrictions have also 
been a factor in this decision, according to the current grantee. 

Recent changes in policy, howevei, have improved the timeliness of naep 
by making the assessment intervals more regular. Reading is scheduled 
for assessment every 2 years and other content areas have been put on a 
4-year or 6-year cycle. There are several technical advantages to this 
change; for example, students at different grade levels can be 
contrasted. 



Timeliness of Reporting There have been recent attempts to report naep results in a more timely 

fashion. Further, efforts to disseminate results were recently enhanced 
throug,'i the development of additional nontechnical products. A particu- 
lar example is the "NAEPgram" recently developed by ets, the grantee, 
as a means of informing the educational community of assessment 
results. ETS reported mailing 100,000 copies of the first "NAEPgram" to 
all elementary and secondary school principals and other professionals. 

In an additional attempt to facilitate the dissemination of findings and 
improve public access to naep information, nie developed the National 
Assessment of Educational Progress Information Retrieval System 
(NAEPms). This computerized data base contains findings and descrip- 
tions of assessments of 9-, 13-, and 17-year-old students, allowing the 
users to tailor their assessments (for example, to examine specific sub- 
groups or unreported naep data). The department reported that 4,000 
copies of the data base had been put into cii culation by May 1986. 



The technical adequacy of naep has been highly regarded in the educa- 
tion community and it has improved over time. In several instances, 
technical advances have resulted in increased relevance at state and 
local levels. Standardized age definitions coupled with alterations in the 
assessment cycle now make it possible to examine differences between 
groups in a given subject area. Sampling and reporting by grade level (in 
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addition to age level) makt it possible to link naep results more closely 
to state and local assessments ('v.id school practices) that are typically 
associated with grade levels. 

Although earlier assessments employed matrix sampling of assessment 
areas, the new naep design employs a variation of matrix sampling 
called ''balanced incomplete block spiraling/* which allows for assess- 
ments of relationships between content areas and background variables 
yet keeps assessment time per student to a minimum (thereby reducing 
the respondents' burden). This alteration in test administration, in com- 
bination with advances in measurement (for example, using item 
response theory), has improved the ability to compare and interpret the 
flndings (through scaling procedures) over time and between groups. 
Although NAEP has generally been viewed in a positive light, department 
officials have expressed concern over the appropriateness of some of 
the interpretations of the data resulting from special analyses.^ 

In terms of some indicators of naep quality, school response rates for 
1984 reading assessments uniformly high (ranging from 84 to 90 per- 
cent). The sample sizes have been enlarged for each age group, increas- 
ing NAEP*s overall precision (although the assessment cycle was changed 
from 1 to 2 years). With regard to sampling variability, technical 
changes meant that the most recent reading assessment had to report 
estimated standard errors and provide extensive caveats regarding their 
level of accuracy. 

Nonsampling errors are reduced because the data collection process is 
standardized through the use of hired and trained staff. All information 
is collected by field staff, who maintain comp arability. Further, unlike 
previous assessments that employed a paced audiotape to provide direc- 
tions to the students while they took the test, in the most recent assess- 
ment (reading and writing) the students were given oral instructions. To 
assess the influence of this change in testing, special data collection and 
analysis documented differences attributable to the changes in testing 
procedures. What is commendable about this research is that it system- 
atically examined the comparability of important changes in procedures. 
It also points up the importance of preserving some of the basic method- 
ology of past assessments in order to maintain the usefulness of data on 
trends. 



^In a 1986 letter to the NAEP contractor, the assistant secretary for OERI criticized the contractor's 
interpretation of NAEP data on bilingual education. 
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Impact The fourth in(licatx)r of quality refers to the use of the information that 

is gathered. In 1976, we concluded that naep's results had to be made 
more useful to state and local policymakers. We based our conclusion on 
responses to a survey in which 71 percent of the local education agency 
ofHcials indicated that they had not used naep concepts, methods, or 
material. Further, while 90 percent of the state education agency offi- 
cials said they had reviewed naep information, 44 percent of those who 
responded rated the concepts, methods, and material moderate to lim- 
ited in their utility. In commenting on our report, officials within the 
department acknowledged the need to improve the usefulness of naep 
and provided a plan for implementing our recommendation. 

Sebring and Boruch conducted a more recent review of the uses of naep, 
finding numerous instances of state and local use.^ They reported that 
12 states had replicated naep completely as part of their own statewide 
assessments and that 14 states had adapted the naep model. Seventy 
percent of Minnesota's districts participated in the "piggyback pro- 
gram," in which local districts work under contract with the state to 
conduct local assessments. Similar practices were found in Connecticut. 
However, Sebring and Boruch noted that not all the uses to which naep 
or NAEP-like studies have been applied meet acceptable research 
practices. 

Overall, in ranking the usefulness of naep to different audiences, Sebring 
and Boruch concluded that naep is the most useful to those with a 
national perspective. However, the capacity to transfer naep's methodol- 
ogy enhances its utility at the state level and below. Currently, naep 
practices are being considered for use in redesigning the elementary and 
secondary education data system maintained by ces and in efforts by 
the Council of Chief State School Officers to obtain comparable meas- 
ures of achievement across states. 



Summary and Conclusions Although naep has provided data on the nation's children since 1969, 

recent changes have helped enhance naep's role in understanding the 
condition of education in the United States. Our review of the available 
evidence suggests that as naep is currently structured, it ranks rela- 
tively high on all four indicators of quality (technical adequacy, timeli- 
ness, relevance, and impact), naep continues its tradition of providing a 



^P. a. Sebring and R. F. Boruch, "How Is the National Assessment of Educational Progress Used?" 
Educational Measurement: Issues and Practice , Spnng 1983, pp 16-20. 
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national perspective on the condition of education and has stimulated 
better state and local assessments. 

NAEP has not escaped constraints associated with the shrinking fiscal 
support foi educational information. It has been affected by budgetary 
declines in two ways. The number of target populations was reduced 
from five to three principal groups, and the assessment cycle was 
altered from aimual to biennial, or longer, some content areas being 
assessed at 4- and 6-year intervals. 

Determining the optimum interval between test administrations goes 
beyond the issue of fiscal resources. The ability of naep to record 
changes in performance depends on maintaining short intervals between 
assessments. As an interval increases, the ability to signal changes 
becomes more limited. Further, naep does not assess many groups, 
including students younger than 9 years old. Expanding the substantive 
scope or the target populations will inevitably require more funding, 
further restrictions in the coverage of principal populations, or greater 
reliance on "other" funds (see table III.1). 

Funding reductions appear to be associated with some benefits to the 
overall quality of naep. For example, budget restrictions seem to have 
resulted in the use of more-efficient sampling designs and testing proce- 
dures. Altering the testing cycle from 1 to 2 years has allowed an 
increase in sample size and, therefore, greater precision. The trade-off 
here is clear — although successful adaptations to fiscal constraints seem 
to have resulted in increased technical adequacy, relevance and timeli- 
ness have declined somewhat. 



The CJommon Core of 
Data 



Purpose and Background Within nces, data on elementary and secondary education are gathered 

from several distinct types of survey (naep amoig them) and adminis- 
trative records. This case illustrates the quality of the Common Core of 
Data, which is based mostly on administrative records maintained by 
state education agencies. Several types of fiscal and nonfiscal data are 
obtained from state administrative records, ocd was initiated in 1974 
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"to fill current gaps in policy relevant educational data, to facilitate provision of 
information needed on a recurring basis, and to increase the usefulness of data to 
meet the needs of educational policy-makers at all levels."^ 

cx:d generally collects information annually or on some other schedule. 
The version of ocd during our review consisted of six surveys (see table 
1X1.4). Information on instructional staff, students, expenditures, reve- 
nues, and school characteristics are gathered from schools and local edu- 
cation agencies by a state education agency and transmitted to nces. 

Planning for the system began with four grants to develop the require- 
ments of state and local users of the Ciommon Core of Data. The system 
was field-tested in 1976 and implementation began in the 1977-78 school 
year. It was intended to replace elsbgis. In its original form, ocd con- 
tained more program elements than we give in table III.4. In fact, our 
third case illustration — the Fast Response Survey System— was origi- 
nally developed as part of the early CCD. Other surveys have been elimi- 
nated or scaled back (as discussed in chapter 2) in concert with a special 
task force of the Conunittee on Evaluation and Information Systems 

(CEK). 

Beginning with an education statistics advisory council report in 1957, 
concerns have been raised about the quality of administrative data from 
intermediate sources. In 1985, the department undertook its own inter- 
nal review of the elementary and secondary education data system, 
including CCD. Department offldals asked the committee on national sta- 
tistics of the National Academy of Sciences ( nas) to conduct an indepen- 
dent assessment of the center.® Both internal and external reviewers 
relied heavily on over 50 letters, comments, and other papers that nces 
officials solicited from professional organizations and users and produc- 
ers of NCES data, nces staff and outside consultants also synthesized the 
comments from these reviewers and commissioned analyses on how the 
system should be reconfigured to take into accoimt its problems and 
deficiencies, nces staff have issued a draft report on how to alter the 
system to improve its overall quality. 



^National Center for Education Statistics, Projects, Products, and Services (Washington, D.C.: U^. 
Government Printing Office. 1976. p. 102). 

^The NAS study, issued in September 1986. was quite broad in its charge and included four maj/or 
aspects: to review, describe, and assess data quality and quality-assurance processes, program con- 
tents and services, and the timeliness of data coUectkm and diaseminatkm 
obstruct or hinder NCES in accomplishing its mission. Only part of the study is descril»ed here. 
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The series of reports generated by these reviews substantially agree on 
the general problems of the technical adequacy and usefulness of ocd 
and other surveys in the current system. Many experts regarded data 
stemming from the elementary and secondary education program as 
deficient on one or more of our quality indicators. Evidence for their 
conclusions does not focus exclusively on ocd, however. Closer inspec- 
tion of each quality dimension points out ccd's strengths and 
weaknesses. 



Several assessments raise questions about the relevance of nces data 
collection efforts for elementary and secondary education. One review 
emphasized the lack of comprehensiveness of the entire elementary and 
secondary education information system.^ The reviewers noted that ocd 
was inadequate for answering questions on the relationships between 
student background, processes, and outcomes but that it did contain 
some relevant information on resources. In contrast, a similar, though 
not strictly comparable, analysis in a 1976 nas report suggested that 
ELSBGis, the forerunner to ocd, had been more responsive to the assess- 
ment of inputs, processes, and outcomes, suggesting that ocd had 
deteriorated. 

A 1986 internal department review of the comprehensiveness and avail- 
ability of data across various programs, including ocd, revealed consid- 
erable gaps in the present statistical system. The chief source of 
dissatisfaction stemmed from the system's inability to answer nine fun- 
damental questions about educational input, participation, process, con- 
tent, cost, and outcomes. Looking across levels of ecucation, the 
assessment showed that data were almost nonexistent for preschool 
children and completely absent for persons no longer in school. The only 
area for which ocd provided adequate data concerned the providers of 
educational services. 

One potential strength of ccd was that data were obtained from all state 
and local education agencies. However, information was reported in 
such a way that it could not always be broken down into meaningful 
units (such as local school districts and schools). This has been a rmjor 
criticism of ocd, especially for the school finance data. In some 
instances, data are no longer being gathered. For example, ccd no longer 



'v. Plisko, A. Ginsburg, and S. ChaiWnd, 'Aaseasing Nationa] DaU on Education," US. DeDartment of 
Education, Washington, DC, August 1966. 
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collects information on what each state pays its teachers, a critical 
resource expenditure. 



Timeliness Several recent reviews have concluded that the center's activities have 

been inadequate with regard to timeliness, a problem that dates back 30 
years. For example, a paper commissioned by nces noted that by July 

1985, data on enrollment by grade were no more recent than fall 1983.« 

To provide a better understanding of such deficiencies, we examined the 
age of data on elementary and secondary education reported in the 1980 
and 1983-84 editions of the Digest of Educaticm Statistics (in September 

1986, the latter was the most recent publication). Since the Digest 
reported both OCD information and data gathered in other ways (such as 
surveys) and by other organizations (the Census Bureau and the 
National Education Association), we report our analysis of ocD and non- 
OCD data separately. 

Table 3. 1 shows that the age of data reported in each issue of the Digest 
for both 1980 and 1983-84 ranged from 6 months to more than 10 years. 
Judging from the age of other non-co) data reported by nces and other 
agencies, the delays associated with CCD are not unique. However, in 
1980, other agencies or sources produced a higher percentage of rela- 
tively current data (12 months old or less). Comparing 1983-84 with 
1980 indicates an increase in age, particularly for data generated under 
the CCD system of reporting, bi addition, the 3-year interval between 
Digests (in this instance, between 1983 and 1986) means that the most 
recent information can be even more out of date. Therefore, we concur 
with the critics who regard the timeliness of routinely reported data as 
a serious concern. 



"G. Hall et al, Altcmatives for a National Data Syatem on Etementary and Secondary Education 
(Washington, DC: Cente^ on Educattti/ltatistKs, December 20, 1965) 
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Tabto 3.1: TiM TiflMlinMS Of Data 
Baportad in tha Pigoat of Education 
Slatlatica in 1M0 and 198»44 





% of total data e 


toments 






1980 




1«'13-84 




Months out of date 


CCD 


OthT 


CCl 


Othwr 


12 or less 


0% 


39% 


6% 


6' 


13-18 


33 


22 


9 


24 


19-24 


33 


0 


18 


6 


25-36 


11 


6 


52 


47 


37 or more 


22 


33 


16 


18 


Number of clwiMnts 


27 


18 


33 


17 



Source, Some of the 1980 CCD data are from tfie Elementary arKl Secondary Education General Infor- 
mation Survey. The remainder of the data are from CCD 



One reason for these reporting lags is that the review and publication 
processes require more time now than in previous years. For example, 
the 1986 Digest was submitted for review in August 1985 but not pub- 
lished until 13 months later. In contrast, the draft Digest for 1964 was 
submitted for review in June and published 3 months later. One official 
noted that reducing the length of the review process could reduce the 
lag between data collection and publication of the results, especially 
since the 1986 document was changed very little by the review. 

A second reason for the publication lags is the timeliness of state report- 
ing. It takes longer now than in previous years to obtain the information 
required from the state education agencies. We were not able to deter- 
mine the reasons for this, but pragmatic steps might be to establish cut- 
off dates and to use estimates for delinquent slates. 



Technical Adequacy As we noted above, cod is composed mostly of data derived from state 

administrative records. The system was designed to provide a census of 
schools and local and state education agencies. The accuracy of the data 
and their comparability across state education agencies is of central con- 
cern for this type of information. Our review reveals limited evidence on 
changes in the quality of cciHlerived data We focus on the availability, 
comparability, and accuracy of selected data elements. 



Availability Whether infv^rmation frorri administrative records can be reported to the 

department dopeads on whether and how they are maintained by state 
education agenaes The most recent and complete assessment of these 
issues as they pertain to cod was conducted by the State Education 
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Assessment Center. This work, known as the Education Data Improve- 
ment Project, was supported by the C!ouncil of Chief State School 
Officers and noes.® State by state, the project examined the comprehen- 
siveness and comparability of selected data elements, some of which 
were part of ceo while others were deemed important enough to be 
added to ocd. 

The Education Data Improvement Project showed that the states differ 
substantially in the availability of data elements. For example, all the 
state education agencies that participated in the study can report enroll- 
ment or membership data on public school students but only 80 percent 
(including the District of Columbia) can report similar data for nonpub- 
lic schools. 

To gain a better understanding of this diversity, we examined tho per- 
centage of states that maintained each of 35 data elements at the school 
district level. Twelve data elements were part of ocd and the remainder 
were identified by Education Data Improvement Project staff as ele- 
ments important enough to include in ocd. Since the states can differ in 
the level of aggregation they maintain for each data element, table 3.2 
displays the frequency of data elements available at the school level of 
aggregation: only 2 of the CCD data elements are available at the school 
level for 40 or more of the states, but 1 1 of the 23 proposed data ele- 
ments are available at the school level for 30 or more states. 



Table 3.2: Data Eleimms Available at the ^^^^^^^^B^^^^^^H^^^^H^^^^^B^^H^^^^^H 

School Level of Aggregation Number of states CCD f>ropose<l 



40-49 2 1 

30-39 6 JO 

20-29 2 5 

10-19 2 4 

0-9 0 3 

lOtal 12 23 



Source The school universe file of the Common Core of Data for 48 states and the District of Columbie 
reported in Council of Chief State School Officers, Summary State Collection Practices on Universe 
Data Elements (Washington. D C U S Government Printing Office, September 1986) 



^*The goals of the Education Data Improvement Project are to descnbe state collection of CX^D data 
elements, describe other elements that might make it more adequate and appropriate for reporting on 
the condition of the nation^s schools, and recommend to CES and the states ways for making it more 
comprehensive, comparable, and timely. 
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Comparability Data Oi. schools and local and state education agencies must be compar- 

able to be useful. Critics of ccd have argued that the data are not com- 
parable because definitions of variables differ within and across states. 
For example, nces and others note that school attendance is defined to 
include ''excused absences" in California but not in other states. 

The Education Data Improvement Project assessed, in detail, the similar- 
ity of definitions and procedures for eiux)llment, fall eiux)llment, mem- 
bership, and average daily membership. A comparison of state 
definitions and procedures with those prescribed by nces showed that 
many of the states that collect these data elements are consistent in 
their definitions of **eiu-ollment'' (27 of 32 states, or 84 percent), "mem- 
bership" (40 of 40 states, or 100 percent), and ''average daily member- 
ship" (40 of 40 states, or 100 percent). In contrast, most of the states 
(44) maintained date labeled ''fall eiu-ollment," but few (only 17, or 39 
percent) agreed with a common definition. The "fall eiu-ollment" defini- 
tions differed in the date used to establish eiu:x)llment (spanning from 
either September or October) or in criteria (different numbers of days 
that had to pass before taking the count). Many of the stetes that agreed 
on definitions of the various date elements differed in the procedures 
they used for calculating them. To explain these and other stete-to-stete 
differences, the project's staff observed that nces is often inconsistent in 
the use of terms on date collection forms and in the guidelines for com- 
pleting them. 



Accuracy Precise estimates of the degree to which ccd elements are in error are 

difficult to obtain. Although nces planned in the early 1980's to develop 
a program of quality-control studies of the date in its collection (similar 
to that conducted by the Bureau of i ^ Census for its current population 
survey), comprehensive assessments were not carried out. Reviewers of 
NCES activities have illustrated technical problems by making selective 
comparisons that may not represent all elements of the ccd date base. 
They have found 

• estimates of dropout rates that differed by 50 percent, 

• estimates of school discipline problems that differed by a factor of 10, 
depending on the source, 

• vocational education enrollments in some stetes that exceeded their 
entire high school populations, and 

• estimates of the size of the population of students with limited profi- 
ciency in English that differed by as much as 200 percent. 
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These examples illustrate some serious inaccuracies, but assessments of 
their prevalence within CCD have not been undertaken, although a 1985 
assessment conducted as part of the Projections of Education Statistics 
program within nces provides some indirect evidence. The accuracy of a 
projection is determined by the adequacy of the projection methods 
(NOES used methods developed by the Bureau of the Census) and the 
consistency of the base data (drawn from ocd) over time, so that its 
analysis provides a partial basis for evaluating the degree of error in the 
data. This may not help detect reporting biases that may persist from 
year to year, but differences between projected and reported values pro- 
vide some evidence of the magnitude of year-to-year instabilities and 
other errors. Flaws in the projection methodology will also contribute to 
such differences, but for short-run forecasts, inaccuracies in the data 
used in the projection are likely to contribute more to the projection 
errors than is the projection methodology. 

nces has been making projections of student enrollment, instructional 
staff, degrees awarded, and expenditures for elementary, secondary, 
and postsecondary education since the mid-1960's. In 1985, nces staff 
assessed the accuracy of their 1966-82 projections by examining how 
closely earlier projections resembled data reported later for those same 
years. For example, enrollments predicted for 1980 were compared to 
the actual enrollments in 1980. For this assessment, multiple sets of pro- 
jections (1 to 10 years) were examined, and the average absolute per- 
centage of projection error was used to assess the general accuracy of 
the NCES projections for enrollment, instructional staff, and degrees 
awarded. 

With short forecast horizons (1 to 2 years), primary and secondary 
school enrollment projections were in error by less than 1 percent as 
seen in table 3.3. Projection errors were higher for number of high 
school graduates- 1 to 2 percent for short forecast horizons — and less 
than 2 percent for instructional staff. Table 3.3 also shows that th<»se 
are considerably less accurate as the forecast horizon increases, espe- 
cially for 10-year projections. Although indirect, the short forecast pro- 
jections suggest that for some variables, inaccuracies might be fairly 
small even if all the errors detected were the result of problems in the 
CCD data. 
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Tnble 3.3: Absolute Percentage Error of 
Projections for Selected Elementary and 
Secondary Education Data by Lead Time 



Data element 



Forecast horiion in years 



1 



Enrollment by grade 



10 



K-8 



K-12 



0.3% 0 6% 0 9% 8 8% 
~72 



02 



04 



0.8 



9-12 



Number of high school graduates 



06 



08 



1 1 



2.1 



Number of classroom teachers 



20 53 
43 124 



09 



1 5 



3.7 



4.7 



Source Adapted from National Center for Education Statistics, Projections of Education Statistics to 
1992-1993 Methodo:ogic a l Report with Detailed Projectio n Tables (Washington. D C U S Government 
Printing Office. 1985). p n 



C€D provides most of the basic data for elementary and secondary edu- 
cation for the Projections of Education Statistics ; it is featured in The 
Condition of Education and the Digest of Education Statistics (about two 
thirds of all tables reporting on elementary and secondary education in 
recent editions have involved data from CCD); and it appears to be used 
extensively by the department's statistical information office in 
responding to questions from a wide variety of federal, state, and local 
policymakers, teachers, and other constituents. However, in serving the 
needs of education policymakers, CCD has not had the kind of impact it 
could have had if problems of technical adequacy, timeliness, and rele- 
vance had been corrected. 



Summai y and Conclusions Although ccd has some strengths with regard to its relevance, timeli- 
ness, and technical adequacy, the balance of evidence suggests that it 
contains numerous inadequacies. With few exceptions, data were not 
uniformly available from all states. When data were available, they 
were not on the same level of aggregation; some states had data availa- 
ble at the school level, while others maintained them at the school dis- 
trict level. Further, definitions and procedures for reporting data 
elements differed across states. With respect to CCD's impact, there is 
little direct evidence of its ase in policy decisions oth'^ • than its role in 
supporting center publications and education projections and as a 
resource for answering inquiries. Despite complaints dating back 30 
years, recent reviews indicate that few of the problems have been 
solved. However, some CCD data were technically sound enough to yield 
consister^ short-term projections, and some data elements were reasona- 
bly cons- stent across states, suggesting that it is possible to obtain some 
usable data from administrative records. 
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Fast Response Survey 
System 



Purpose and Background The Fast Response Survey System was established by nces in the mid- 

1970's to furnish data quickly when timely national estimates were 
needed for important educational issues, frss was designed to (1) mini- 
mize the respondents' burden (typically three to five questions are 
asked in a sample survey), (2) keep the time between a survey in the 
field and reporting its results to a minimum through a network of data 
coordinators, and (3) collect narrowly limited information that was not 
available from other sources. 

FRSS was designed to gather information as needed through a contractual 
arrangement with a private survey research firm from one or more of 
the following six educational sectors: (1) state education agencies, (2) 
local education agencies, (3) public elementary and secondary schools, 
(4) nonpublic elementary and secondary schools, (5) institutions of 
higher education, and (6) noncoUegiate postsecondary schools with 
occupational programs. A data collection network was developed for 
each sector. C!oordinators assisted in coP^ct'ng data by maintaining liai- 
son with sampled institutions and agencies. Representatives of each 
institution or agency were identified and responsible for completing the 
questionnaires. Data collection was intended to take 6 to 10 weeks. 

Whereas all state education agencies were included in the system (mak- 
ing it a cer'^us), stratified random samples (with numbers of respon- 
dents ranging from 500 to 1,000) were designed to yield reliable national 
estimates for schools, local education agencies, and other institutions. 
Twenty-four reports or bulletins were issued between 1976, when the 
first FRSS study was reported, and September 1986. In interviews with 
present and former frss project monitors, we were told that surveys of 
state education agencies usually cost about $25,000 each and surveys 
based on nationally representative samples cost $80,000 to $100,000. 
The FRSS system is currently funded by a 5-year contract and has an 
annual budget of about $200,000 to $350,000. 



Relevance fkss is different from other information collection syt^tems by being an 

information service , available only to department officials, rather than 
an existing information source . Practices differed across frss surveys, 
but the contents were generally specified collaboratively by the 
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requester, center staff, the Committee on Evaluation and Informa- 
tion Systems, and the contractor, all jointly involved in refining the 
uolicy questions, developing the survey, and determining the nature 
and scope of the analyses. The system was designed to tailor data 
collection to the ne ^ds of the requester. That is, relevance is built 
into the system, if it is fast enough to deliver results before changes 
overtake the requester and the questions. 

FRSS differs from other information-gathering activities in terms of 
whom it attempted to serve in each survey. As a matter of policy, the 
system is limited to officials within the department who have a high- 
priority need for quick information. Our analysis of the initiation of 23 
requests showed that three broad groups have relied on frss. In 10 
instances, requests came from officials within the Department of Educa- 
tion (the assistant secretary, undersecretary, or a program officer 
within the department). In 4 other cases, the department official used 
the system as a way of fulfilling parts of congressional mandates or 
requests. Six studies were initiated by special commissions, members of 
advisory groups such as ncer and the National Commission on Excel- 
lence in Education, or leaders of special initiatives established by the 
secretary or the president. It was not possible to determine who initiated 
the remaining 3 frsS reports. 



Timeliness frss was developed to address, in part, concerns for which existing data 

were not available, not current, or not national in scope. While evidence 
of the timeliness of all fast response surveys is incomplete, it appears 
that delays have gotten longer. Figure 3.1 displays the available data on 
the elapsed time between the completion of a survey protocol and its 
publication or release date. For the early years of the system, 1976-79, 
the publication date could be determined for only five of the nine 
surveys. For these, the elapsed time varied between 6 and 14 months. 
The data for later years, 1981 to 1986, are more complete; the elapsed 
time was variable, ranging from 6 to 34 months. Timeliness appears to 
have declined, on the average, since the late 1970's. 

Interviews with center staff suggest that this indicator overestimates 
and underestimates the timeliness of the frss products. On the one hand, 
since no documentation is available on when requests were made, using 
time when the survey protocd is developed and has been cleared 
through the review process— that is, through the Federal Education 
Data Acquisition Council (fedac), ceis, and the Office of Management 
and Budget (omb) — may underestimate the elapsed time by 4 to 8 
months. 
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Figure 3.1 : FR8S Time Between Survey Development and Publicatiuii 1976-85" 

Number Months 



Number for which 0 5 10 15 20 25 30 35 
of data are ■ ■ 1 1 1 i i 1 
surveys missing 1 i 1 1 1 1 1 1 


1976 1 
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3 3 
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' " represents an individual survey 
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Figure 3.2: FRSS Time Between Survey and Earliest Known Release Date 1976-85* 

Number Months 

Number for which 0 5 10 15 20 25 30 35 




' represents an individual survey. "A" represents an early or advanced release 
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On the other hand, preliminary results were frequently reported to a 
requester prior to the official publication date. Although the informa- 
tion on preliminary releases is incomplete, the surveys for which early 
release dates are known show little increase across time (see figure 3.2). 
With the exception of one relatively recent survey (frss number 19), 
elapi^ ed time ranged between 3 and 8 months. 



Technical Adequacy We examined the technical adequacy of each frss survey in light of 

information presented in publicly available documents — that is, final 
reports and bulletins. To measure the technical adequacy of these prod- 
ucts, we examined the sampling frame: precision, or sampling error; 
response rates; and treatment of nonresponse and other forms of non- 
sampling error. 

Sampling Frame A sampling frame is the basis for the selection of a sample. Whether a 

sample represents the population depends on the accuracy of the sam- 
pling frame. In most reports, the population of interest was clearly iden- 
tified. However, the extent to which the operational sampling frame 
represented the population of interest was assessed in only one survey 
(FRSS number 22). In one other (frss number 10), the data base used for 
drawing the sample of higher education institutions was 5 years old. 
Although the number of schools that were no longer in operation was 
reported, the universe may have onutted new openings after 1975. In 
another instance (frss number 2), decennial census data had to be used 
to stratify local education agencies, although this information was more 
than 5 years out of date by the time the report was issued. 

Response Rates For the 22 reports available for our review, the median reported 

response rate was 95 percent — by conventional survey standards, a 
remarkably high rate. For state education agencies, the median response 
rate was 99 percent. However, closer inspection of the information pre- 
sented for some sample surveys suggests that response rates may have 
been inflated by the way the rates were defined. While the initial and 
final sample sizes were usually reported and decisions to exclude sample 
elements (as when some sample elements were not appropriate) were 
briefly docimiented, response rates were not reported consistently. In 7 
cases, response rates were calculated after nonparticipants (that is, 
refusals) had been excluded from the total usable sample size. In 8 
instances, the rates appeared to be based on conventional practices, but 
in the remaining 7 cases too little information was presented in the 
reports to make a detennination. The description of methodological 
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practices was less comprehensive in the later years, when the ten- 
dency was to issue bulletins rather than final reports. 

Precision of Estimates Four of the surveys we reviewed were targeted at all state education 

agencies. Since they represented a census, a consideration of variation 
from sampling error was not necessary. In all the remaining sample 
surveys, the reports described the precision of the estimates by provid- 
ing standard errors of estimate or coefficients of variation. Statistical 
decision rules were described in all applicable cases. 

Nonsampling Errors The credibility of survey data depends also on the absence of other 

types of error not related to sampling and referred to collectively as 
''nonsampling errors." For example, bias can be introduced when 
selected local education agencies refuse to participate (do not respond). 
Although most reports described adijustments for nonresponse, analyti- 
cal details wen, too sparsely reported to ascertain the adequacy of these 
adjustments. Since the level of participation was uniformly high, the 
influence of nonresponse was miniir ized. 

Other forms of response error can influence the results. In several 
reports, the adequacy of the data that were obtained was described. For 
example, in one survey of institutions of higher education, the respon- 
dents were asked to provide estimates if they could not rely on actual 
administrative records in answering certain questions. Separate and 
combined analyses were reported. In this instance, the estimated data 
underreported student-retention rates relative to the data derived from 
actual records. In another report, considerable effort was devoted to 
understanding and resolving the reasons the survey produced results 
that were not consistent with federal records. In terms of survey prac- 
tices, this level of quality control can be considered exemplary. Also, 
new quality-assurance procedures have been initiated, including the 
appointment of a chief statistician who will be reviewing all reports. 

Impact Confirming how requesters used the data produced from frss surveys 

was beyond our scope, noes documents asserted that frss surveys were 
used to inform policy and to guide program directions. At least one 
assessment of the center identified dissatisfaction with frss. In a 1978 
management evaluation of nces conducted by the department, a pro- 
gram official within the department reported that two attempts to use 
FRSS were imsuccessful. According to the report, the center refused to 
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conduct one survey because of the sensitivity of the proposed ques- 
tions; the second request went unanswered because nces "took too 
long to agree to run the survey."*® In a follow-up interview with a 
former frss project officer, we were told that about one third of all 
requests are denied for two reasons. First, to justify clearance with 
OMB, the topic has to have a high priority within the administration 
or a mandate from the Congress. Second, since the frss funding level 
is limited, the number of surveys that can be conducted is limited. 



Summary and Conclusions fRSS has been called upon to produce information through surveys 

nearly two dozen times over the past JO years. On the whole, these 
surveys appear to have responded to the information needs of the 
requesters. Although timeliness appears to depend upon how it is 
deflned, the incomplete evidence available suggests that the practice of 
issuing preliminary results minimizes delays. On the technical side, these 
surveys appear to follow generally accepted practices, although the 
reporting of actual procedures is often sparse and some practices could 
be improved. Given the relatively low budgets associated with each sur- 
vey, technical adequacy appears to be satisfactory. 



Across our three cases, quality was variable, naep received generally 
high marks for relevance, timeliness, technical adequacy, and impact. 
CCD's marks were low on all four indicators, frss got high marks on rele- 
vance, medium on timeliness, and medium on technical adequacy; our 
information was too limited to assess its impact. Thus, it is possible to 
obtain high-quality information through department support and man- 
agement; it is also possible to have msuor, long-term problems in a signif- 
icant information activity. Timeliness and technical adequacy may be 
more difficult to achieve than relevance and impact, at least in the three 
statistical systems we examined. 

In the next .sections, we examine influences on quality in these 
instances. In the next chapter, we examine influences on both produc- 
tion and quality, drawing on these examples and on other information. 



Relevance As we have noted, relevance refers to the extent to which an informa- 

tion source answers questions posed by the requester or others. Looking 



^"U S Department of Education. Office of Management Analy.SLS and Systems. "Mana|;ement Evalua- 
tion National Center for Education Staastics," Washington, D C . 1978. p 12 
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across the three case studies, we see that increases in relevance have 
been associat with the addition of data elements, contrary to current 
initiatives to reduce the respondent's burden. Relevance has been low- 
ered by reductions in population coverage and content areas and the 
lengthening of the time between assessments. These actions have been 
necessitated, in part, by budgetary constraints. In addition, the case 
studies illustrate that tailoring assessments and dissemination to meet 
the needs of primary sponsors and other users enhances relevance. 



Expanding Information The high marks that naep received on relevance have been linked to its 

Collection Activities recent expansion of information on student background and school char- 

acteristics. The original purpose of naep was to serve as a benchmark 
for assessing national progress in education, but as experience with naep 
increased, greater demands were placed on it to provide data that would 
allow additional analyses to explain variations across states and levels 
of performance. With encouragement from the Congress and evidence 
from the public reviews it has received, naep has attempted to be 
responsive to these criticisms by adding several hundred data elements 
on student characteristics and attitudes and school environment. Simi- 
larly, FRss represents new data collection efforts, frss has achieved its 
high level of relevance by filling specific gaps in information needed for 
policy purposes. In both cases, relevance has been increased by adding 
data collection activities. 



Reductions in Information Not surprisingly, reductions in information with respect to population 

coverage, content areas, and frequency of assessments — principally in 
response to budgetary constraints— have reduced the relevance of naep. 
Furthermore, while ccd has been criticized for its lack of relevance for 
answering questions about who is served in public and private schools, 
staf fmg, and resources and for examining the relationships between 
educational inputs, process, and outcomes, efforts within the depart- 
ment have been directed at reducing the amount of information collected 
through CCD. Even frss, which is intei ded to provide policy-relevant 
information that is not available elsewhere, has had to turn away 
requests. One reason is limited resources. Another reason is the need to 
demonstrate high priority or a mandate to justify clearance through 

OMB. 

The three cases also show that whether an information system is judged 
relevant may depend on how circumscribed the request is in terms of 
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contents and for whom it is collected. In this regard, the clearest case of 
high relevance is frss, for which typically a single requester has a nar- 
rowly constrained set of questions and studies are tailored to the partic- 
ular interests of the sponsor. It ^pears that a key to the success of frss 
is that its structure optimizes the collaboration of the requester, center 
staff, and contractor responsible for collecting the data, naep and CCD 
have to serve many needs of diverse policymakers, increasing the likeli- 
hood that some questions may not be sufficiently answered. 

Additional support for the importance of tailoring studies to the needs 
of particular constituents can be seen in nie*s mandated 1974 compensa- 
tory education study. By several accounts, this study is a successful 
instance of the production of high-quality information relevant to the 
congressional oversight process. Althou^ a host of factors account for 
its success (for example, it was well funded and timely), congressional 
staff had worked closely with nie staff before the mandate was pro- 
posed and throughout the execution of the study. In other words, given 
a limited number of constituents, substantial collaboration, and enough 
resources, this example, like the frss studies, suggests that the informa- 
tion needs of policymakers can be adequately addressed from currently 
available methodologies. 

Flexible DLssemination As our cases indicate, statistical information systems are often required 

to serve the needs of many constituents, naep and department staff 
have devised several noteworthy practices to increase the relevance of 
data produced by naep. In particular, naepirs public-use data t£q)es have 
been developed to facilitate secondary data analysis, allowing research- 
ers to answer questions that were not directly addressed in naef's for- 
mal publications. Judging from the number of copies of naepirs in 
circulation, this data set is a considerable national resource. 

CCD has been less well regarded in this area. While the center has main- 
tained public-use tapes on elementary and secondary statistics, the last 
time they were listed in nces Program and Plans was in 1980. Further- 
more, although nces sponsored efforts in the early I970's to make data 
available through the Education Statistics Information Access System, 
this system appears to have been eliminated when nces reduced dissemi- 
nation services. 



We noted earlier that timeliness can be thought of in two ways: the time- 
liness of assessment and the timeliness of reporting and disseminating of 
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Other information products (for example, technical reports, bulletins, 
and public-use data tapes). 



Timeliness of Data Collection Recent changes in the assessment schedule for naep have made the 

intervals between assessments for each content area more regular, but 
information in some content areas is substantially out of date. Given 
time now required in the development of adequate survey and assess- 
ment instruments and the clearance process, data may be too old to be 
useful, unless they are acceptable to those who are using them. For some 
users, of course, frss may be a reasonable "fail-safe," if issues arise that 
require data sooner than can be obtained through ongoing activities. 

Timeliness in Reporting All three activities have been cited as having had problems in the timeli- 

ness of reporting with many of the same consequences as the delays in 
collecting information. However, the solutions are numerous. For exam- 
ple, within CCD, data are gathered by state education agencies and may 
not be submitted on schedule. This is less likely to happen in surveys 
like those of naep or frss, in which the contractor has control over the 
data collection schedule. Also, delays in the review and publication pro- 
cess have been noted in all three cases. For example, frss has avoided 
this problem by providing the requester with preliminary results and 
accompanying them with bttle explanatory material. The new "E.D. 
Tabs" reporting format extends this practice to data from other 
surveys. Other plans that are being developed within the center include 
the publication of early estimates. 



Technical Adequacy Technical adequacy refers to the appropriateness of the design and exe- 

cution of a study, given the questions that have been posed. In addition, 
a technically adequate study does not contain serious flaws and is 
reported in sufficient detail to allow the readers to ascertain the credi- 
bility of the data upon which the results and conclusions depend. 

The three cases show substantial variability in their technical adequacy. 
The differences can be organized around three interdependent practices: 
(1) data collection methodology, (2) quality-control procedures and the 
adequacy of reporting on the strengths and limitations of the data, and 
(3) the extent to which assessments of data quality are conducted. 
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Data Collection Methodology naep and frss rely on representative samples (except for frss surveys 

directed at the universe of state education agencies) as their primary 
means of obtaining national estimates for policy-relevant questions. CCD 
relies on state administrative records. The technical adequacy of naep 
and FRSS was superior to that of CCD. The reasons for the differences in 
quality between survey-generated data and data obtained from adminis- 
trative records are fairly obvious. Both surveys are adnunistered by a 
grantee or contractor and use standard data collection instruments, min- 
imizing the kind of error introduced by noucomparable data elements, as 
in CCD. 

The FRSS and naep examples also show that practices vary according to 
the purpose of the data collection effort. For example, both use 
probability surveys (except for the census of state education agencies), 
but the precision of the estimates derived from them differs. Data 
derived from frss surveys are considerably less precise than NAEP-gener- 
ated data because of differences in sample sizes. Given the differences in 
cost of these two types of survey, this is to be expected. 

The cases also show that some technical difficulties and practical con- 
straints are inevitable. For example, even though naep is a carefully 
planned survey, employing state-of-the-art sampling techniques, esti- 
mates of precision are approximated, geographic comparisons are lim- 
ited to regional breakdowns, and results are sensitive to the test 
administration procedures. In other words, the improvements in some 
practices associated with naep will not resolve all problems, and in some 
cases they create new technical difflculties (for example, the modified 
matrix sampling technique increases efficiency but cannot be used with 
paced audio instructions). 



Quality-Control Procedures The technical adequacy of information can diminish at each phase of the 

data collection process. To counter this, various quality-control proce- 
dures are generally used. However, it is not possible to eliminate errors 
completely. Gtenerally accepted research standards acknowledge this 
and advise that strengths and weaknesses should be examined and 
described. 

Of the three activities, naep uses the most comprehensive set of quality- 
control procedures. For example, assessment materials for the most 
recent naep were reviewed by external consultants for bias; new mate- 
rial was tested, reviewed, revised, and retested prior to final selection; 
sampling frames were verified for accuracy; and schools were visited to 
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verify that data collection was undertaken as planned. Much of the high 
regard naep received for its technical adequacy derives from these qual- 
ity-control procedures. 

Performing quality-control checks is time consuming and r quires ade- 
quate resources. Depending on the purpose of the study, this level of 
effort may not be appropriate for small-scale surveys like those con- 
ducted under FRSS. For FRSS, the quality-control procedures were less 
extensive. Survey questions were reviewed by center staff, representa- 
tives of CEis, and omb reviewers. According to a former project officer, 
this type of review avoided serious ambiguities in question wording, 
bias, and unnecessary respondent burden. As we have indicated, the 
precision of estimates was considered where appropriate and proce- 
dures were used (although they were not always well described) to 
account for other sources of bias (such as nonresponse). In general, 
quality control within frss appeared to be satisfactory, considering the 
relatively low budget associated with each survey. 

Our review of ccd suggests indirectly that quality control was not suffi- 
cient, especially during the early phases of data gathering. Available 
evidence from the Education Data Improvement Project study shows 
that standard definitions were not adhered to in reporting data, infor- 
mation was not umformly available across states, and estimates of the 
accuracy of data were generally not available for most data elements 
within CCD. The absence of well-documented quality-control procedures 
has threatened the credibility of cCD-generated data 

Another aspect of quality control is expert review. Since its mandate, 
NAEP has been developed under the surveillance of a standing panel of 
representatives from business, industry, and the general pubic as well as 
education practitioners, who meet up to three times a year to guide its 
purposes, specify domains to be examined and measured, examine sam- 
pling strategy, and review analysis and reporting plans. In addition, the 
legislation for naep requires a review at least once every 3 years and 
provides for public comment on how it is conducted and its usefulness. 
Although not all the recommendations resulting from these reviews 
have been followed (for example, recommendations on funding), much 
of the technical guidance has been followed. While it is not possible to 
separate the influence of the review panels, the caliber of contractor 
staff attracted to naep and selected through the panels of still other 
experts, and the technical and managerial skills of staff assigned to 
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monitor naep, it seems fair to attribute some of the high marks on tech- 
nical adequacy to the panels and reviewers, who do sustained and regu- 
lar work. 

In contrast, CCD has been reviewed as part of general oversight of the 
center. Over the past 20 years, there have been several reviews, such as 
the 1986 review by the National Academy of Science's conunittee on 
national statistics. The concerns raised by the panels tend to be peren- 
nial. What they criticize has tended not to get fixed, although technical 
problems such as noncomparability of data elements could be corrected. 
Commendably, center staff responsible for the projections of educa- 
tional statistics assessed the accuracy of their projection methods and 
assumptions in 1985. And the technical adequacy of CCD infonnation 
was examined in a 1986 study of the State Education Assessment 
Center. Such technical self -assessments are exemplary but obviously 
differ from a full technical review. 

FRSS has not received external reviews. Relatively speaking, survey 
research for brief, targeted questions is a well-established technique. 
That the system has functioned as well as it has with regard to technical 
adequacy (where this can be judged from available information) illus- 
trates that the need for external reviews varies with the technical com- 
plexities of the activity. 



Page 66 



68 



6A0/PEMIV8a4 The Production and QuaUty of Education InfonnaUoii 



Chapter 4 



Influences on Production end Quality, Agency 
Comments, and Our Response 



In examining influences on the production and quality of information, 
we drew on our findings from the analyses of awards, the three case 
examples, budget analyses, interview:; with department officials, and 
prior studies of education informatior.. We believe that ..aring the period 
of ou^ s^udy, the primary influences on the information that was pro- 
duced and its quality were funding, mandates, changed priorities, and 
expert reviews. Other influences were congressional action other than 
mandates to collect information and changes in the leadership of the 
information-producing units. 

More specifically, we found that the msyor influence on information pro- 
duction was severe reductions in funding levels. Information quality 
was directly and indirectly affected by funding levels. Congressional 
mandates played a notable role in information production — those activi- 
ties that did not carry mandates were most vulnerable o funding 
declines, ^ hanges in priorities were linked to rapid changes in leader- 
ship. Furt^ reviews by experts contributed to quality. During the 
period of r study, information production was also affected by the 
Paperwork Reduction Act and its interpretation and by shifts in priori- 
ties associated with changes in leadership. Any factor alone was gener- 
ally but not always influential. Results were clearest when appropriate 
levels of funding, congressional guidance, expert reviews, and leader- 
ship priorities were all present and working in the same direction. 



Because decreases in awards have affected information production, we 
examined the fiscal resources available for them. To assess changes in 
the size of the federal investment in education information when the 
federal budget was generally being reduced, we chose two benchmarks: 
(1) the change in the federal mvestment in education channeled through 
the department and (2) general changes in research, statistics, and eval- 
uation. For the first benchmark, we assume that not every dollar of ser- 
vice should be matched by a penny or a dime for information but that 
the capacity to obtain information about education should increase or 
decrease somewhat as overall education expenditures increase or 
decrease. Therefore, we asked not only "How has the federal investment 
in education information changed over time?" but also "Has the invest- 
ment been differentially affected by the overall o --containment 
efforts?" The second benchmark examines difl /e.ttial effect relative to 
agencies with similar mission^ 

Our analysis of the trends in fiscal resources in 1973 to 1986 for nie, 
NCES, and opbe shows that the investment declined Furthermore, reduc- 
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tions for education information were substantially greater than those 
for education as a whole.' Fiscal resources for educational research, sta- 
tistics, and evaluation were cut more than in agencies with similar mis- 
sions between 1980 and 1984. 



The Department of Figure 4.1 shows that fiscal resources for the department increased in 

Education as a Whole current dollars from approximately $6.1 billion in 1973 to $ 19.5 billion 

in 1986, an increase of 220 percent — 38 percent in 1972 dollars. In con- 
trast, the trends for fisca] support of the production of research and 
statistical and evaluative information are quite different. 



Figure 4.1: Department of Education Obligations in Current and Constant 1972 Dollars for Fiscal Years 1973-86' 
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'Includes the Office of Planning. Budget, and Evaluation, the National Center for Educational 
Statistics, and the National Institute of Education. Constant 1972 dollars are conr<puted by 
using the implicit price deflator for federal government purchases of goods and services as 
reported in Survey of Current Business. 

Source: Selected issues of Survuy of Current Business for 1979-86; Department of Education 
data; and Office of Management and Budget. The Budget of the United States Government: 
Appendix (Washington. D.C.: 1975-86). 
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^Trend analyses began and ended for NCES in 1974 and 1975; for OPBE in 1986 and 1985. 
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Research For nie, figure 4.2 shows that cunent dollar amounts fluctuated over 

the period 1973-86 but the general trend was downward. That is, in 
1973, NIE had current dollar obligations of roughly $ 107 million; by 
1986, these resources had fallen to $51 million, a 52-percent decrease. 
When viewed in real terms, the trend depicted in figure 4.2 is even more 
dramatic: from 1973 to 1986, nie experienced a 79-percent reduction in 
fiscal resources, despite the 38-percent increase in the overall federal 
investment in education.^ 




1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 19*^5 1986 



Current 
Consta't 

•Constant 1972 dollars are computed by using the Implicit price deflator for federal 
government purchases of goods and services as reported in Survey of Current Business> 

Source' Selected issues of Survey of Current Business for 1979-86: National Institute of 
Education data; and Office of Management snd Budget, The Budget of the United States 
Government - Appendix (Washington. D.C.: 1975-81 and 1983-85). 
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^In fiscal V '>T 1986, OERI was reorganized. This may account for some of the reduction in resources 
for research, but between 1973 and 1985, resources for research had already been reduced by 79 
percent. 
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Viewed from our second benchmark — changes in general support for 
research across the federal government — the funding level for nie has 
fallen short of the support provided elsewhere. While the^real purchas- 
ing power of overall federal research funds, excluding education, grew 
by 3.7 percent between 1980 and 1984, nie's funding level dropped by 
48 percent. During tliis period, the overall federal investment in educa- 
tion declined by 10 percent in constant dollars. 



Statistics Figure 4.3 charts similar information for nces. The trend is more erratic, 

but the net result is roughly the same. In both current and constant dol- 
lars, fiscal resources for nces declined.^ Current dollar obligations in 
1974 were $11.8 million; in 1986, the figure was $8.7 million, represent- 
ing a 26-percent decrease. In real terms, resources for statistical activi- 
ties declined from about $10 million in 1974 to about $3.6 million in 
1986, a 65-percent reduction. 

Further, nces suffered greater losses in funding than other statistical 
agencies. In a study of federal statistical programs, the Congressional 
Research Service reported that fiscal year 1984 budgets for seven msyor 
federal statistical agencies, including nces, were 18-percent lower over- 
all in real terms than the fiscal year 1980 budgets. If we consider only 
nces, however, we find that the inflation-adjusted budget — including 
salaries, expenses, and program funds — decreased from $14.9 million in 
1980 to $10.7 million in 1984. This is a 28-percent decline for nces alone. 
The 1980-84 budget cuts were, therefore, disproportionately large com- 
pared not only to the general decrease of 10 percent for education in 
real terms but also to cuts in other federal agencies involved primarily 
in statistical activities. 



^The decline in the late 1970's can be parUy explained by the congressionally mandated transfer of 
NAEP from NCES to NIE in March 1978. NAEP remained the resp nsibility of NIE unUI 1985, when it 
was transferred to CES in the reorganizl^lo^f OERI 
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1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 198S 1986 



Current 
Constant 

*NCES began operation in 1974. Constant 1972 dollars are computed by using the implicit 
price deflator for federal government purchases of goods and services as reported in Survey of 
Current Business. 

Source: Selected Issues of Survey of Current Business for 1979-86; National Institute of 
Education data; and Office of Management and Budget, The Budg^.t of the United States 
Government: Appendix (Washington, D.C.: 1978-86). 
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Evaluation For the Office of Plaiming, Budget, and Evaluation, our assessment is 

limited to funds for contract activities from 1975 to 1985.* As shown in 
figure 4.4, funding activities peaked in 1978 at $28.8 million and 
declined thereafter. The most dramatic decline was between 1980 and 
1982. Over the 10-year period from 1975 to 1984, the decline was 31 
percent in current dollars. In 1972 constant dollars, this represented a 
64-percent decline. 



Rgure 4.4: OPBE Obligations in Current ^^^^^^^^M 
and Constant 1972 Dollars for Fiscal 

Ysars 1 975-85 30 doNvs in MHHont 




1975 1976 1977 1976 1979 1980 1981 1982 1983 1984 19S5 



Current 
■laiaai Constant 

Source: Offtce of Planning, Budget, ard Evaluation annual evaluation reports 1975-85 



Our second benchmark was derived from our January )87 report on 
federal evaluation activity, in which we showed that resources for pro- 
gram evaluation declined by 37 percent between 1980 and 1984 across 



"^Figures are from the contract listings in the annual evaluation reports We checked the figures for 
each fiscal year except 1985 against the annual evaluation repot for the subsequent year Since the 
1986 report was not av ' ble, we used 1984 data rather than the unverified 19a5 figures in our 
calculations of changes -ume 
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federal nondefense agencies '^ During this period, the office showed a 63- 
percent decline in purchasing power for program evaluation activities.^ 

For research, statistical, and evaluative information, the patterns of 
decline in funding were consistent and precipitous. Further, the consis- 
tency of decline in resources across these three types of information 
suggests across-the-board reductions in information rather than a sub- 
stitution of research for evaluative data or of statistics for either 
research or evaluation. Funding support for the production of all three 
general forms of information about the condition of education has 
decreased more than 60 percent in real terms since the early to middle 
1970's. 

For all three forms of education information, major declines between 
1980 and 1984 ranged from 28 to 63 percent in 1980 constant dollars. In 
all three cases, the declines exceeded those observed for agencies and 
departments with similar missions. The greatest discrepancy, however, 
wat' for the research function; while federally supportei i research 
rece ved, in general, a small increase from 1980 to 1984, funds for nie 
were reduced by nearly half. This change led to the decrease in awards 
repo sed in chapter 2. 



The case studies show that fiscal resources influence quality. For exam- 
ple, in 1976, NAEP reduced the number of skill areas it routinely assessed 
from 10 to 5. When the assessment schedule was altered to accommo- 
date budget cuts in 1980, increased demands were placed on naep staff 
to assist state and local agencies in the use of naep material, naep bud- 
gets have been supplemented by assessments conducted for other units 
within the department (see table III.l). 

Reviews of naep have been critical of sonie of these changes, suggesting 
that skill areas should be expanded, assessments should be more fre- 
quent, and scope should be broadened to younger students, for example, 
or to higher-order cognitive skills. To accommodate these concerns, 
funding would have to be adUusted upward— that is, the proposed 



Fiscal Influences on 
Quality 
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''t* S G€. i*.»ral Accounting Office, Federal EvaJuation Fewer Units, Reduced Reso urces, Different 
Studies 1 rom 1980 , PEMD^7-9 (Washington, D C January 23, 1987) 

^'EsUmates differ on OPBE's funding level Figures from OPBE's review of a draft of this report indi- 
cate a 28-percent decline m current dollars, or a 45-percent declme in constant dollare OPBE based its 
figures on data obtained from other sources in addition to the annual evaluation reports 
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changes are simply beyond the current capacity of the information-gath- 
ering activity. 

Our review suggests that other forms of support that are indirectly 
affected by resources, such as technical assistance, influence the quality 
of information. Reviews of naep show that providing technical assis- 
tance to states and local areas has made information more relevant. And 
flexible dissemination formats (such as naepirs) have been linked to 
increases in timeliness, relevance, and the impact of naep. The services 
provided by fuss have also been shown to fill information gaps in a 
timely fashion. Of course, these support services divert funds away 
from other information-gathering activities. 

For data collected within CCD, the evidence on the utility of support ser- 
vices and technical assistance is mixed. In examining the components of 
the pre-ccD data system, we found that nces had devoted 5 staff years 
annually to developing and updating a series of handbooks that pro- 
vided common deflnitions for education-related terms and data require- 
ments under ei^bgis. This aspect of the nces portfolio was eliminated 
when staff assigned to this function retired. The last revision of the 
flnancial accounting handbook was published in 1980. Another compo- 
nent of pre-ccD assistance from nces was on-site visits and workshops to 
assist state and local personnel in completing information requests. As 
fiscal resources within nces declined, field visits were curtailed. 

Although the early CCD experiences with technical assistance and other 
types of support have been unsatisfactory and have not appeared to rid 
CCD of persistent problems, we did find evidence in another department 
activity that federal assistance can have some positive effects on the 
quality of data reported from state and local education agencies — 
namely, the Title I Evaluation and Reporting System. At first, participa- 
tion in TIERS was voluntary, and 5 years were devoted to implementing 
the system. Its use was mandated in the 1978 Education Amendments. 
TIERS ultimately produced data that were comparable in format across 
states, but the level of technical support, research and development, and 
state capacity-building was substantial, costing about $37 million 
between 1975 and 1981. That is, quality w^s improved but it took time 
and money. 



Resources dlld Support ^^'^ regard to the direct effects of fiscal resources, one seemingly obvi- 
^ t^t^ corrective action would be to restore funding levels. Our three case 
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Studies suggest that reductions in funding can influence aspects of qual- 
ity but that funding must be examined in relation to the purposes of the 
data collection activity. Quality seems to be related not to dollars alone 
but, rather, to the match between required activities and required funds. 

For example, the three case examples differ considerably in fiscal sup- 
port. NAEP, a m^r activity within the center, corrju^aided over $6 mil- 
lion in resources annually. CCD began in 1976 with $900,000 annually 
and was reduced to about $400,000 by fiscal year 1986; funds were 
withdrawn in fiscal year 1987. frss is operated at about $200,000 to 
$350,000 per year. Our review of quality shows that naep and frss, the 
most and least costly, received moderate to high marks for relevance, 
timeliness, and technical adequacy while ccd seems to have faltered 
under various levels of funding. 

Furthermore, our review suggests that despite losses in purchasing 
power, NAEP has increased in technical adequacy and improved one 
aspect of its relevance by expanding its collection of background infor- 
mation on students, their attitudes, and school conditions, frss, despite 
its rather low level of expenditure, appears to have maintained an 
acceptable level of timeliness, relevance, and technical adequacy, 
although not all frss requests have been granted, because of budgetary 
constraints and other issues. 

In contrast, ccd has not received sustained support. In 1987, state edu- 
cation agency data-gathering went unfunded. The consequences were 
clearly demonstrated in our review of the data that have been gathered. 
Some data elements appear to be consistent with noes specifications, 
consistent over time (as indicated by small projection errors), and com- 
parable across many states, but the system has been vigorously criti- 
cized with respect to its overall quality. 

The criticism hardly seems surprising, cod appears to have been 
underfunded for its purposes and expectations. In the mid-1 970's, the 
department requested $10 million to $23 million for ccd, a fraction of 
what one study estimated would be necessary. Funding never exceeded 
$900,000 per year— less than $20,000 per state agency. Judging from 
the discrepancy between proposed and actual funding, the idea of hav- 
ing a common core of data that could be used by all relevant policymak- 
ers was not adequately implemented. 

These c?5es show that while it is necessary to provide a stable source of 
funds, it may not be necessary to increase fiscal resources in order to 
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enhance the relevance, timeliness, and technical adequacy of informa- 
tion. Rather, funding levels should match expectations for the nature of 
the data to be collected and the intended uses of the information. 



Ck)ngressional 
Mandates 



In addition to establishing general statutory missions for information- 
producing units, the Congress has used speciflc legislation, conference 
reports, and hearings as ways of requesting information about the con- 
dition of education, naep, for example, is a congressionally mandated, 
ongoing activity, tiers and the National Vocational Education Data Sys> 
tem (nveds) were mandated and intended as ongoing information sys- 
tems. The Ck>ngress has also asked for special studies and surveys. 
Examples include the 1974 nie title I compensatory education study and 
the 1985 survey of teacher supply and demand. 



Congressionally mandated activities can have a variety of influences on 
information-pnxlucing units. First, ongoing mandates can provide direc- 
tion for the aUocation of staff and resources. Their increasing influence 
has approximated management by legislation, particularl> in nie. As 
table 4.1 shows, about 79 percent of me's resources for research in 1984 
went to legislatively required activities such as naep, eric, and laborato- 
ries and centers, in contrast to 55 percent in ] 980. The requirements 
have typically not specifled the research questions to be answered 
(except in terms of broad areas of emphasis) but have often directed 
operating procedures for the awards in the federal agency responsible 
for the study and funding authority. 



Table 4.1: National inttitute of Education 
ReMarch Obligations in Fiscal Ytars 
1M0andlM4 



Activity 


1M0 


1964 


Mandated* 


55% 


79% 


Discretionary 


45 


21 


Total awards 


$73,625,000 


$38,795,000 



Includes National Assessment of Educat.onal Progress Educational Resources InformatKXi Centers, 
an6 the laboratories ar>d centers 

Source National Council on Educa* o(\a\ Research annual reports. Washington D C for fiscal years 
1960 and 1964 



Our review of the role of mandates as a request strategy suggests that 
they can protect an activity by ensuring a sustained level of support, 
even during periods of budget-cutting, but other activities may be 
affected by insuf flcient funding or staffing or both. Our analyses of 
shifts in priorities showed that information-gathering activities that do 
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not carry a mandate were more vulnerable when faced with fiscal 
constraints. 

For example, in the late 1970's, the Bureau of Education for the Handi- 
capped devised a multistudy, longitudinal evaluation plan to assess 
aspects of Public Law 94-142, the Education for AU Handicapped Chil- 
dren Act. One component of this plan, a set of longitudinal case studies 
of the effect of the law, was initiated but was terminated early. Budget 
cuts, changes in priorities, the study design, and the lack of an explicit 
mandate were among the reasons offered for the termination. 

Another type of vulnerability concerns the ability to collect data when 
studies are not mandated. In our study period, at least one frss request 
could not be honored: omb required evidence that the information to be 
collected was requested under the mandated activity, but the proposed 
projects could not be justified on this ground. Further, omb increased its 
efforts to reduce information collection by deleting data elements from 
OPBE evaluation questionnaires on the basis that these data were not leg- 
islatively required. Available instances suggest that unmandated infor- 
mation-gathering activities are vulnerable to changes in policies and 
priorities and to funding constraints. 

Second, specially mandated studies have a large but transient effect on 
the operations of information-producing agencies. They are less predict- 
able than ongoing activities such as naep. Depending on a study's size or 
timing, it can consume a substantial amount of a unit's resources, incur- 
ring opportunity costs with regard to other activities. A unit reduced in 
staff size 16-20 percent may not be able to expand in response to certain 
requests. Further, in some instances, the questions to be answered, time- 
liness, and methods to be used were explicitly stated in the request in 
ways that were difficult to carry out. This is not to say that some man- 
dates may not have a positive influence by altering normal agency rou- 
tine. For example, in one case the mandate directed that reports were to 
be sent without prior department review to the requesting group, thus 
reducing the department's influence on the study design and final 
report. This has been seen as shortening time and increasing relevance 
in the sense that departmental policy concerns and congressional inter- 
ests did not have to be reconciled. 

With regard to quality, naep has produced reports that are viewed as 
timely, technically adequate, and relevant descriptions of the condition 
of education. However, mandates alone are not sufficient to ensure that 
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high-quality information will be available when it is needed. For exam- 
ple, NVEDS was mandated in the Education Amendments of 1976. In 
1983, 0MB disapproved the department's request for data coUection for 
the school year 1983-84, on the grounds of severe technical problems 
with the data. Here, the system was mandated with little consultation 
with the department, no resources were provided, and the time for 
reporting information back to the Congress proved to be unrealistic. 



Expert Reviews of the 
Information-Gathering 
Function 



The mission statements of the three units we examined are general, pro- 
viding considerable latitude for the development of a portfolio of infor- 
mation. To ensure that these functions are being properly carried out, 
different review methods have been tried out, varying the authority 
given to the reviewers (policymaking versus advisory) and the regular- 
ity of review (periodic versus ad hoc). For example, until recently, nie 
was guided by ncer, a policymaking group that was given broad author- 
ity and met regularly. In addition to assessing the nie portfolio of activi- 
ties and reporting to the Congress, it served as a policymaking body, 
setting priorities on dissemination, for example, and on equity-related 
activities. 



From the reports that were issued, it is evident that ncek's review was 
susceptible to factors that may have little to do ^ith the overall quality 
of nie's work. For example, in 1980, ncer provided a detailed progress 
report to the Congress on activities that were highlighted as priorities in 
the Education Amendments of 1980.^ That report also summarized 
available knowledge on ways to improve educational practice. The over- 
all tone of this document was that nie was responsive to priorities and 
taking a responsible approach toward identifying the agenda for future 
research. Two years lat^r, under a new chairperson and with new mem- 
bers, the NCER report covering Hscal years 1981 and 1982 characterized 
nie's efforts as "a flight from reality and from traditional moral val- 
ues.''** In reviewing nie's previous work, the new ncer used as the princi- 
pal evaluative criteria the choice of topics that had received research 
support, finding them objectionable. 



^National Counal on EducationaJ Research. "Sixth Report of the Nanonal Council on Educational 
Research," Washington, DC, fiscal year 1980 

'^National Counal on Educational Research, "Research m Retrospect," seventh annual report, Wash* 
ington, D C . 1982 
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A variety of advisory groups have reviewed the statistical function 
within the department since the mid-1950's. In carrying out their advi- 
sory functions, these groups submitted reports that contained recom- 
mendations on ways of improving statistical activities. In fact, many of 
the problems the National Academy of Science's conmiittee on national 
statistics identified in its 1986 report on noes bear a striking resem- 
blance to problems in a 1957 report issued by an advisory group. Given 
the persistence of the problems, the advisory panels do not appear to 
have been very influential in improving the quality of the data-gather- 
ing activities, despite the fact that early coimcils were composed of 
highly regarded individuals. 

NAEP has been reviewed through a complex process that was built into 
legislation. Since its inception, naep has been guided by a panel of 
experts who have meet regularly and have decisionmaking powers. In 
addition, periodic reviews by experts (at least once every 3 years) were 
mandated. Available evidence suggests these reviews have had several 
positive influences on the contents, operation, and management of naep. 
In particular, one aspect of the mandated review was a stipulation that 
the users of naep be given an opportunity to conunent on its relevance 
and utility. Although many forces were at work in shaping naep, the 
department summarized several comments from the field in its report to 
the Congress that resulted in alterations in naep that have been 
regarded as improvements in relevance and technical adequacy. 

Similar improvements by the regional laboratories and national centers 
have resulted from reviews by one-time panels with advisory powers. 
For example, in the Education Amendments of 1976, the Congress estab- 
lished a 15-member panel of educators to review and report on the labo- 
ratories and centers. The panel members were appointed by the director 
of nie from a field of 450 candidates nominated by organizations and 
associations in the education community. The panel reviewed each labo- 
ratory and center in terms of the potential value of the work that was 
proposed and its national significance, management, rud track record. 
The Congress charged the panel with making recommendations for 
improving and continuing individual laboratories and centers. 

Relative to other reviews, the scholarly manner with which tlie panel 
executed its charge is readily seen in its systematic method. Each labo- 
ratory and center was examined on common criteria, and projects were 
judged on the*^ , elevance to the mission of the center or laboratory and 
on their technical merits. The assessment was balanced, in the sense 
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that the strengths and weaknesses of the proposed projects were identi- 
fied. The panel members articulated rationales for the importance of 
given focal areas, their contribution to education practice, and expected 
accomplishments. A subsequent review initiated by ncer did not assess 
the level of technical adequacy in the laboratory and centers products 
but showed that, collectively, they have contributed information in 
many of the same areas identified by the national reform studies, sug- 
gesting that the laboratories and centers were dealing with issues of 
national importance. 

NAEP, national centers, and regional laboratories are msyor activities in 
the department. They consume a large share of the budgets allocated to 
NCES and nie. Extensive review activities are therefore justified and 
appropriate. For activities such as frss, a different level of review may 
be warranted. With the exception of incomplete reporting practices, we 
judged the overall quality of frss as moderate to high, suggesting that 
elaborate review and external review are not always necessary, frss 
received little formal review besides the routine review of data quality 
by project monitors and through the recent competition of the contract. 
This seemed entirely satisfactory, given the level of funds allocated to 
this activity. 

While expert reviews have positively influenced quality in general, 
there are limits to what can be expected, and quality-control activities 
may have to compete for funds. 



In studies of the information-producing units, different directors and 
changes in senior staff were reported to have notably influenced the pri- 
orities and operations of the units. We were concerned in this report that 
changes in leadership in the education information units could similarly 
influence data production and quality. Therefore, we examined the 
changes and, where possible, attempted to identify their influences on 
education information. 

Each of the information units changed in top management positions dur- 
ing the 1980*s. In order to determine how widespread these changes 
were, we asked the units to provide us with the number of top manage- 
ment positions and the names and tenure of individuals serving in them 
from September 1980 to September 1986. Table 4.2 shows considerable 
turnover in each unit, although the data were not always complete. 
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Table 4.2: Management of Information Units in Fiscal Years 1980-86 




Head of unK 




Tod manaoement 


Unit Title 


Number in 
position 


Number acting 
in position 


Number of 
positions 


Number in Numiier acting 
position in position 


National Institute of Education Director 


4 


3 


5^ 


10 6 


National Center Education for Administrator 
Statistics 


2 


1 


5 


15*^ 2 


Office of Planning, Budget, Deputy underr irv 
and Evaluation 


4 


1 


9 


17 



^Represents a best estimate, one of the five, the associate director for field-initiated and internal stud- 
ies, was notcre-ted until 1984 

^One position, assistant administrator for research and analysis, went unfilled from November 1980 to 
June 1982 and from May 1986 through at least January 1987. when we obtained our data 
^OPBE did not indicate any managers acting in positions 

Source Department of Education. Office of Personnel and executive offices for the Office of Education 
Research and Improvement and the Office of Planning. Budget, and Evaluation 

NiE ad a tx)tal of 7 differ ent directors from 1980 to 1986, 3 of the 7 
serving as acting dire^or. At least 16 persons served in the 5 other top 
management positions, 1 of which was created in 1984. In six cases, 
individuals served on an acting basis or as special assistants. 

The positiOii of Ov^ ministrator for nces was much more stable than the 
position of director of nie. From 1980 to 1986, nces had 2 top adminis- 
trators and an acting director serving for 2 months in the transition 
between them. The turnover in the other statistical management posi- 
tions, however, was similar to what occurred in nie. There were 5 top 
management positions at nces, 17 persons serving in them from 1980 to 
1986. Two individuals served in an acting capacity. 

The Office of Planning, Budget, and Evaluation is he-^ded by a deputy 
undersecretary. One of the 5 persons in this position from 1980 to 1986 
r^erved in an acting role. Seventeen individuals served in the 9 top man- 
agement positions during this period. 

In summary, there have been many shifts in leadership in information- 
producing units since 1980. nces has been the most stable, but all three 
units have had multiple changes in the top position. All three have also 
had multiple turnover in other top management posts, some managers 
leaving in a matter of a few months. Other positions h ive been vacant 
for various periods. 

Staffing levels from roughly 1980 to 1986 are presented in table 4.3. 
During the period we examined, two types of change took place among 
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professional staff: v decrease in the total number of positions and staff 
turnover. In nie, 91 persons were in excepted service on May 15, 1979, 
and 191 were in civil service positions on September 30, 1981. In 1986, 
there were 200 professionals — 39 in excepted service and 161 in civil 
service positions. These changes represent declines of 57 percent and 13 
percent for the excepted service and civil service, resj)ectively.® 





Table 4.3: Changes in Education information Professionai Staff Between Fitcai Years 19«M) and 1986 






Total 




1986 


1980 staff 


1980 staff 








New hires 


remaining 


elsewhere in the 


Unit 


Service 


1983 


1986 


since 1980 


in 1986 


department 


National institute of Education 


Excepted 


91^ 


39 


33 


6 


0 




Civil 


191^ 


161 


114^^ 


47 




Total 




• 


200 


147 


53 


9 


National Cente*" for Education Statistics 


Excepted 


0 


2 


2 


0 


0 




Civil 


123 


96 


50 


46 


19 


Total 




123 


98 


52 


48 


19 


Office of Planning, Budget, and 














Evaluation 


Excepted 


0 


0 


f 


f 


f 




Civil 


32 


27 


f 


f 


f 


Total 




32 


27 


f 


f 


1 



^May15, 1979 
''September 30, 1981 

'^Within the Office of Educational Research and Improvement, 29 new hires were in the office of 

research, 47 were in programs for the improvement of practice, and 38 were in information services 

*^tncludes 4 staff who were assigned to the Center for Education Statistics 

^Since figures are from two different periods, calculating a total woi^ld be misleading 

'Data not available 

Source Executive office of the Office of Educational Research and Improvement and GAO surveys of 
federal program evaluation activities for 1980 and 1984 

The turnover was high, especially in excepted-service positions. Of the 
91 excepted-service employees in May 15, 1979, only 6 persons (7 per- 
cent) were still in the department in September 1986; none of these indi- 
viduals had been reassigned to other units in the department. Of the 191 
professionals m civil service positions, 47 remained and 9 of them (5 
percent) were in positions elsewhere in the department or had been 
reassigned to the Center for Statistics. 



^Excepted-service positions are unclassified civil service positions or those outride the competitive 
serv'ce, excepted from the requirements of competitive service by law, executive order, or commis- 
sion regulation 
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The number of staff ar. nces fell 20 percent from 1980 to 1986: from 123 
(0 excepted service, 123 civil service) in 1980 to 98 (2 excepted service, 
96 civil service) in 1986. Turnover has been significant since 1980; o;ay 
46 of the 123 staff in 1980 still remained at nce? in 1986. These 46 make 
up a little less than half the current number. An additional 19 of the 
1980 staff are now in positions elsewhere in the department. 

For the Office of Planning, Budget, and Evaluation, our dzta are taken 
from our survey of federal program evaluation activities for 1980 and 
1984 and are limited to the evaluation component. Within this unit, 
staffing declined by 16 percent from 1980 to 1984, from 32 to 27, and 
according to the staff, remained at about the 1984 level through 198o. 
There were no excepted-service employees until recently, when 3 were 
hired. Thus, there were many changes in upper management but there 
was little staff turnover at the lower levels, providing some stability to 
the unit. 

In summary, we found overall <?eclines in the number of professionals 
available to carr^ out the information-producing functions. Further, 
although the Office of Planning, Budget, and Evaluation was reportedly 
stable, turnover within the other units suggested less consistency in car- 
rying out information activities and potential pi-oblems with staff 
knowledge of department operations and legal procedures. 

The consequences of management changes can be found in the opera- 
tions and priorities of the information-producing units. For example, one 
priority for identified research at me in 1978 was complex learning 
skills. To develop this new area, nie conunissioned papers to identify 
key questions for further research, and the papers were reviewed in 
1980 in a conference bringing together researchers and practitioners. In 
1981, when the fii jrsnt competition was held, more than 90 proposals 
were received and reviewed by panels of experts, which recommended 
30 for funding. In 1981, however, the turnover of directors resulted in a 
hold on funding. The new director did not regard this area as a priority. 
No awards were made after the grant competition. In other words, the 
cycle for research funding from the initiation of a priority through the 
awards process to reporting the results may take many years, but the 
tenure of the director is typically less than a year— long enough to stop 
what was started but not long erough to see initiatives to completion, 
except where they are protected by congressional mandates, such as in 
the laboratories and centers and naep competitions. 



'^Some awards for research on coi iplex learning were eventually made under later NIE directors 
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Congressional actions other than mandates for certain tj-pes of informa- 
tion have influenced the production of information, as when fedac was 
established in the 1978 Education Amendments to eliminate excessive 
detail and unnecessary information requests. In 1979, fedac began its 
reviews of education data collection in the Department of Health, Edu- 
cation, and Welfare. Within fedac's first 9 months of operation, the data 
burden was reportedly reduced by almost 13 percent (a reduction of 
about one million burden hours). 

In 1980, tne Congress passed the Paperwork Reduction Act (Public Law 
96-511), whose key objective was to ensure that information requested 
by federal agencies was needed by an agency, unavailable elsewhere, 
and efflciently collected. The act appears to have had a substantial 
effect on the volume of paperwork required for federal operations. By 
fiscal year 1984, the department reported reductions of 35 percent from 
the 1980 base in reporting requ?rements and paperwork. 

As we illustrated in previous examples, strict interpretation of 
paperwork reduction manoates has led to approval of data collection for 
some progran^iS only when detailed legislative demands for specific data 
elements can be identifled. We did not independently assess the extent 
to which low-quality or duplicative data collection was halted as a result 
of these acts and reduced an unnecessary data burden. 

More direct reductions in information-gathering have been imposed in 
specific program legislation. For example, reporting and evaluation 
requirements for the msjor federally funded compensatory education 
program ware m^de inapplicable with the passage of the Education Con- 
solidation and Ir.provement Act of 1981 (Public Law 97-35). This act 
amended previous legislation in an effort to "eliminate burdensome, 
unnecessary, and unproductive paperwork and free the schools of 
unnecessary Feder?* supervision, direction, and control" (section 552). 
Further, the Deficit Reduction Act of 198^1 (Public Law 98-369) set 
targets for savings in federal government operations. Areas in the act 
relevant to information-gathering include staff travel, the use of consul- 
tants, and publications. 



Paperwork Reduction 



Agency Comments and 
Our Response 



The Department of Education agreed in general with the findings cited 
in a draft of Wis report. It believed our report will perform a valuable 
function by documenting a long-term decline in resources for research, 
statistics, and evaluation. (Its letter is reproduced in appendix IV.) How- 
ever, the department raised three main concerns. First, while it 



1 The Productioii and Qiullty of Edocatioii InfonBAdOB 



Caiapter4 

Inflaences on Production and Qiudity» Agency 
Coiiimento» and Our Response 



acknowledged the validity of many of the points raised for the period 
covered by the report, it believed that in recent years the department 
has "taken clear and decisive action to address most of the problems 
cited in the repori " In particular, it noted that since 1985 each of the 
three information-producing imits has been reorganized or its lines of 
authority and responsibilities have been altered. According to the 
department, the report does not accurately reflect the current situation. 

Second, the department disagreed with our analyses of shifts in priori- 
ties. Specifically, it argued that increases n the proportion of funds allo- 
cated for dissemination are a positive step toward improving the utility 
of research, not a threat to new data collection, as we claimed; it 
asserted that there has been a significant degree of consistency and con- 
tinuity in research priorities and that areas of study that we claimed 
were not being funded are currently supported under awards to labora- 
tories and centers, "minicenters,'* an "urban superintendents' network," 
an intramural research project, and research grants. 

Third, the department disagreed with our assessment of the implications 
of shifts in who is producing educational information and how its pro- 
duction is funded. The department agreed that funding patterns have 
limited its flexibility fcr determining who is fimded but pointed out that 
10 new awards were issued as part oi the fiscal year 1986 field-initiated 
grants competition and that since fiscal year 1986 seven oeri fellow- 
ships have been filled by scholars, researchers, and practitioners. The 
department also argued th&c its procurement methods are an attempt to 
foster, rather than constrain, creativity and "invite alternative strate- 
gies and fresh ideas." 

In its comments, the department also presented an extended discussion 
of the current situation, provided addition?! budget figures, and offered 
further documentation on the number and types of activities for years 
not covered ia the draft report. 

With respect to the department that our findings do not reprtoent the 
department's current situation, we acknowledge that changes have been 
initiated since 1985 in the structure and op\^ration of information-pro- 
ducing units. With regard to opbe and ces, we explicitly mentioned many 
of the topics raised in the department's comments. Several other actions 
the department referred to were initiated as recently as March 1987 and 
could not have been incorporated into our assessment, since our data 
coUecti., A extended up to fiscal year 1986. Since we were unable to ver- 
ify all new data that the department provided in its letters, we have not 
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altered our .ext or observations, except where it has been feasible to 
note updated information. A substantial proportion of the department s 
letter is devoted to describing current and planned activities. 

It is too early to determine whether the organizational and procedural 
changes that have been initiated will adequately address the problems 
we identifled in this report or new ones that the changes might create. 
Oiu" analyses were based on the most current information available at 
ti I time of our study, and, for the most part, new data still are not 
available to assess the effects of the actions the department reported. 
None of the problems we reported occurred overnight, and many of 
them have been longstanding. Thus, while it is useful to have the addi- 
tion?l information on the department's recent efforts, sustained atten- 
tion will be needed to improve the status of education information. 

Our report could serve as a partial baseline against which to assess the 
effects of departmental initiatives to improve the quality, relevance, and 
timeliness of education information. In reviewing the department's com- 
ments, hov/ever, we found no mention of any plans to assess progress 
empirically. It is too soon now to measure the effect of recent depart- 
ment efforts, but making formal plans for evaluation would be a most 
timely endeavor. 

The department's second concern involved several points about our 
analyses of shifts in priorities. The department seems to have miscon- 
strued our central point about the shift in emphasis toward service-ori- 
ented activities — notably dissemination — at the expense of new data 
collection. We did not, as the department contends, distinguish dissemi- 
nation from research. Rather, we distinguish service-oriented activities 
like dissemination from new data collection . Further, while we agree 
that dissemination is a fundamental part of the research process, in light 
of the dramatic reductions in Hscal resources, more for dissemination 
means that there can only be less for new data collection. Dissemination 
can usefully remain a critical part of the research process only if the 
data that are being disseminated are relevant and timely. If resources 
'vere to decline further, there would be less new information to dissemi- 

.iA:, Herein lies the threat that increased emphasis on dissemination 
I yses for now data collection. 

Tne department made two additional points about changes in priorities. 
It took issue with our example relating changes in leadership to changes 
in priorities. We stated in the report that the consequences can be felt in 
both tiie operation of an^h nriorities for information-producirg units. Our 
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data show that the instance we cited was not unique. Changes in 
research priorities associated with shifts in top leadership are acknowl- 
edged in the seventh annual National Council on Educational Research 
report to the President and the Congress. The chairman of ncer stated 
that since 1981 nie's leadership has attempted to move away from the 
ifforb* of its predecessors and, by 1983, changes in research priorities 
were perceptible from a review of new grants. 

Further, the department argued that funds were not available to permit 
significant changes in priorities, since the Congress restricted an 
increasing percentage of the nie budget for institutional awards. We dis- 
agree. Our data in chapter 2 show that the resources that remained after 
mandated activities were funded were spent in very ditferent areas in 
1980 and 1985, our years of focus (see table 2.9). 

The department also objected to our analysis of changes in areas of 
investigation, citing activities directed at gathering information on 
**many of the most important questions and issues in education today '' 
(emphasis added). Forming centers that address education reform issues 
is an important step in addressing these contemporary issues for educa- 
tors and policymakers. However, this step does not make up for the 
years when these and other topics were emerging as critical issues but 
no work was being done. Some way for keeping in step with the areas 
that are emerging and currently important is needed in education. Plan- 
ning efforts should include review by experts to reduce gaps between 
the information that exists and the issues being dealt with by educators 
and policymakers. In other words, the department needs to ensure that 
**fforts will be made to identify new and emerging issues and to imple- 
ment information-gathering on these issues. 

The department's third concern was with our assessment of the conse- 
quences of changes in who is producing information and how producing 
it is funded. Our analyses show an increase in the use of contracts and a 
decline in the number of awards issued to individual researchers. With 
regard to procurement practices, we acknowledge that nie used irany 
funding mechanisms to accomplish its work, including grants, contracts, 
purchase orders, and interagency agreements. However, the data we 
reported m chapter 2 indicate that the m^rity of the later work was 
funded by contracts. The department asserted that contracts are u? .-d 
when it wants to ensure that the substance of the work is clearly articu- 
lated and to provide for an appropriate level of accountability. How- 
ever, we think that other consequences go along with using contracts, 
such as the type of review described. Furthermore, while it is admirable 

S,9 
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to extend invitations to researchers to suggest alternative approaches — 
thereby fostering creativity — it must be remembered that the scope of 
the work (for example, the questions to be answered) is generally speci- 
fied in requests for proposals, leavinf; less room for the imaginative 
researcher. 

With regard to the reduction in the number of individual awards, the 
department noted that the fiscal year 1986 field-initiated research pro- 
gram resulted in 10 awards. Compared with 1984 and 1985, when no 
awards were made (see table 2. 1), this is clearly an improvement, but it 
still represents an 83-percent decline from the number of unsolicited 
proposals awarded in 1980. Therefore, we believe that while there have 
been some signs of restoring this aspect of the department's information 
portfolio, fewer opportunities for "fresh ideas" from the field are avail- 
able new than in the late 1970's and early 1980's. 
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TaM U: Random S«iiH>te Of NIE ActivftiM for 


Activity 


nocipioni 


Cost 


Period coverod 

by award Type 


Study area 


Funding 
mactian^ 


To examtne microprocessing 
technology in schools 


Bank Street College, New 
York. N Y- 


$80,764 


9/80-6/81 New data 
collectK>n 


Computer 
technology 


Contract 


Research on the psychological 
process underlying reading 
comprehension 


Camegie-Melton 
University. Pittsljurg. Pa 




collection 


Reading and 
writing 


Grant 


To assess cognitive 
development of high school 
geometry students 


University of Chk:ago. 
Chicago. Ill 


$49,908 


6/80-8/81 New data 


Math and 
science 


Grant 


To identify the influerx^e of 
nfx)thers. parenting skills for 
preschool children 


Vert>al Interaction Project 


$62,124 


collection 


Parents and 
family 


Grant 


To tmpfove plans for tn-serv»ce 
teacher education in San 
Marcos through research 


Southwest Educational 
Oevek)pment Lab. Austin. 
Texas 


$27,000 


9/80-9/81 New data 
collection 


Teachers 


Contract 


To find ways to inaease Huron Institute, 
evaluation productivity for Cambridge. Mass . and 
dedsionnftaking and cross-study Arlington. Va. 
management 


$101,250 


5/80-5/83 New data 


Miscellaneous Contract 


To inform state-level education 
polk:ymakers and others about 
recent finance devek)pments 


Educatkxi Comm.ssion of 
the States. Denver. Cok) 


$247,137 


6/80-5/81 Dissemination 


School 
finance 


Contract 


To survey and assess the 
literature and theory of the 
private organizations of 
teachers 


Institute for the Study of 
Contemporary Social 
Problems, Seattle. Wash. 


$11,786 


6/80-9/80 Synthesis 


Teachers 


Grant 


To study the organizatkxi of 
magnet schools 


University of Wisconsin. 
Madison. Wise. 


$38 046 


12/79-12/80 New data 
collection 


Desegregation Grant 


To conduct and provide 
technk^al assistance 


Dingle Associates, 
Washington. D C. 


$63,043 


12/79-9/80 Technical 
assistance 


Dissemination 


Contract 


To support expenmental 
activities that denrxxistrate 
ways of increasing numbers of 
minorities arxj women in 
advanced educational research 


MALDEF. San Francisco. 
Calif 


$78,000 


9/80-9/81 Demonstration 


Minonties 


Grant 


ERIC Qeartnghouse for Junior 
Colleges 


University of California. 
Los Angeles. Calif 


$202.47" 


10/80-10/81 Dissemination 


ERIC 


Contract 


Dissemination capacity- 
buikJtng grants 


Indiana Department of 
Publk: Instruction. 
Indianapolis, Ini 


$79,812 


5/80-5/81 Dissemination 


State 

capacity- 

building 


Grant 


To provkje training and 
supportive services to NIE's 
educatkxi policy fellows 


George Washington 
University. Washington. 
DC 


W0.650 


9/80-8/81 Training 


Education 
policy 


Grant 


Historical inquiry of teachers' 
perceptk>ns of their profession 
1840% 


Lynn CadwallaJer. 
Amherst, Mass 


$9,680 


8/80-1/81 Data analysis 


Teaching 


Grant 



•Fifteen actrvities were randomly selected from a list of 476 items in the ^980 NCER annual report 
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Tabto 1^ Random Sampto (rf NIE ActivitiM for Fi^ 


Mr 1985* 








ActMty 




Cost 


Period covered 

by award Type 


Studv area 


Funding 
mechaniafn 


Preplanning activities for new 
teaching and reading centers 


Harvard University, 
School of Education. 
Cambridge, Mass 


$530 


Not listed Planning 


Teaching 


Purchase 
order 


Micfocomputers arxj literacy 


Harvard University, 
Cambridge. Mass 


♦o^T con 


9/30/Ov^9/29/86 New data 
collection 


Computers 


Grant 


Children's early understanding 
of numbers 


Camegie-Melkxi 
University, Pittsburgh. 
Pa. 




12/1/83-11/30/86 Nev -^ata 
CO' on 


Mathematics 


Grant 


Using research knowledge to 
improve teacher education 


George Mason 
University. Fairfax, Va. 


$31.80^ 


Not listed Dissemination 


Teaching 


Contract 


Using research krx>wledge to 
improve teacher education 


Utah State University 


$23,150 


Not listed Dissemination 


Teaching 


Contract 


Teacher educaion 
demonstration program 


University of Georgia. 
Athens. Ga. 


$450 


10/1/85-9/39/86 Dissemination 


Teaching 


Contract 


Th-^ role of schools in education 


Mk^higan State 
University. Institute for 
Pesearch on Teaching. 
East Lansing. Mk:h 


$218,755 


9/30/83-3/30/86 New data 
collection 


Education 
standards 


Contract 


Sixth annual area semirws for 
graduate students 


Courtesy Associates. 
Washington, D C. 


$13,422 


2/14/85-6/1/85 Dissemination 


Seminars 


Contract 


A study of targeting practices 
used in the ct^ter 1 program 


SRA Technok)gies. 
Mountain View. Calif. 


$750,000 


8/1/85-3/31/87 New data 
collection 


Disadvantaged 


Contract 


Research dissemination through Source Telecomputing 
telecommunications Corp.. McLean. Va 


$19,800 


3/1/85-3/1/86 Dissemination 


Department 
communication 


Purchase 
order 


ERIC document reproduction 
service 


Computer Mk:rofilm 
Intematkxial. Arlington. 
Va. ^ 


$442,779 


6/29/79-4/30/85 Dissemination 


ERIC 


Contract 


State technology leadership 
project 


Council of Chief State 
School Officers. 
Washington. D C. 


$129,707 


7/1/85-6/30/86 Dissemination 


Education 
Technology 


Contract 


Second lab review meeting 


Dingle Associates. 
Washington, D C. 


$39,030 


3/22/85-6/1/85 Competition 
panel 


Laboratory 
review 


Contract 


Technology task force writers* 
meeting 


Courtesy Associates. 
Washington, D C 


$7,640 


Not listed Meeting 


Education 
technology 


Contract 


Expert witness in office of civil 
rights enforcement action 


Robert Calfee. Stanford 
University. Stanford, 
Calif 


$19,572 


9/12/84-3/31/85 Witness 


Civil nghts 
enforcement 


Purchase 
order 



Vifteen activities were rancomly selected from a list of 188 items in the '985 NCER annual report 
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Tabto lA: Random Sampte of OPBE ActtvWet for Hsc»f Year 1980" 



Activity 


RecipMfit 


Fiscai year cost 


Period covarad 

by award Typa 


Study area 


Study ct parental involvement in 
various legislatively mandated 
programs 


System Development 
Corp.. Santa Monica. 
Calif 


1978 $639,158 

1979 $1,309,423 

1980 $60' 310 


9/78- 


-4/81 


New data collection 


Parents 


Analysis of issues in ESEA title 1 
.valuation and reporting 


RMC Research Corp . 
Mountain View. Calif 


1978 $398,755 

1979 $259,104 

1980 $320,315 


7/78- 


-4/81 


Issue analysis 


Disadvantaged 


Study of ttie use and effects of 
alternative measures of 
comparability 


Applied Urt>anetics, 
Washington, DC 


1979 $200,000 

1980 $177,140 


9/79- 


-G/81 


New data collection 


Disadvantaged 


Operation of ESEA title 1 
technical assistance center, 
region 1 


RMC Research Corp , 
Portsmouth. N.H 


1979 $676 '13 

1980 $882,992 


10/79- 


-9/81 


Technical 
assistance 


Disadvantaged 


Operation of ESEA title 1 
technical assistance center, 
region VIII 


American Institute for 
Research, Palo Alto. 
Calif 


1980 $1.660,71 5 


10/79- 


-9/81 


Technical 
assistance 


Disadvantaged 


Operative ESEA title 1 technical 
assistance center, region VIII 


Northwest Regional 
Laboratory, Portland, 
Ore- 


1979 $708,200 

\\XU 9mf jOf 


10/79- 


-9/81 


Technical 
assistance 


Disadvantaged 


State refinements to ESEA title i 
evaluation and reporting system 


Rhode Island 
Department of 
Education, Providence, 
R-l 


1980 $33,245 


12/79- 


-2/81 


State capacity- 
building 


Disadvantaged 


State refinements to ESEA title 1 
evauaiion ano reponing sysiem 


Arkansas Department 
OT cuucaiion, Liiiie 
Rock. Ark 


1980 $30,174 


11/79- 


10/80 State capacity- 
building 


Disadvantaged 


State ref incn>ents to ESEA title I 
evaluation and reporting system 


Pennsylvania 
Department of 
Education, Harnsburg. 
Pa 


1980 $39,327 


1/80- 


-3/81 


State capacity- 
building 


Disadvantaged 


Support of the higher education 
panel 


National Science 
Foundation, 
Washington, D C 


1975 $63,442 
1975 $69,900 

1977 $70,000 

1978 $77,190 

1979 $77,250 

1980 $141,262 


7/74- 


-9/81 




Higher education 


Assessment of the 
Strengthening Developing 
Institutions Program 


Research Triangle 
Institute. Durham, N C 


1980 $127,561 


9/RO- 


-9/82 New data collection 


Higher education 


Assessment of the tSAA TV 
program by examining its 
production, distribution, and 
financing 


Abt Associates, 
Cambridge. Mass 


1977 $87,986 

1978 $185,277 

1979 $180,491 

1980 $6,882 


9/77- 


-1/81 


New data collection 


Desegregation 



(continued) 
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Activity 


Recipient 


IHscal year cost 


Period covered 

by award Type 


Study area 


Evaluation of title 1 of the Library 
Services and Construction Act 


Applied Management 
Sciences. Silver 
Spring. Md 


1978 $405,500 

1979 $154,600 

1980 $25,862 


9/78-1/81 New data collection 


Miscellaneous 


ExploratOiy evaluation of follow- 
through service projects 


Applied Management 
Sciences. Silver 
Spnngs. Md. 


1980 $29,873 


9/79-12/80 New data collection 


Disadvantaged 


Field readers, small purchase 
orders, printing, travel, etc. 


Miscellaneous 


1980 $89,326 


10/79-9/80 Support 


Miscellaneous 



*AII activities were funded by contract Activities associated with a particular fiscal year often received 
funding in other fiscal years as well Fifteen activities were randomly selected from a list of 1 19 items 
shown as funoed or ongoing in fiscal year 1980 in the annual evaluations report for fiscal year 1980 
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Tabto 1.4: Random Sample of OPBE Activities for Fiscal Year 1985" 


Activity 


Recipient 


Rscal year cost 


Period covered 

by award Type 


Study area 


National longitudinal evaluation 
of the effectiveness of services 
for language-minority, limited- 
English-proficient students 


Development 
Associates, Arlington, 
Va. 


1983 $1,514,000 

1984 $2,619,352 


12/1/82-12/30/86 New data 
collection 


Language minority 


Addition of Native American 
students with limited English 
speaking ability to the national 
longitudinal evaluation 


Development 
Associates, Arlington, 
Va. 


1985 $438,591 


9/17/85-12/16/86 New data 
collection 


Native Americans 


Assess ECIA chapter 1 grant 
program for the handicapped 


Research and 
Evaluation Associates, 
Chapel Hill, N C 


1985 10,000 
1984 263,091 


10/1/84-12/31/85 New data 
collection 


Handicapped 


A study to document the 
process and procedures that 9 
states will develop to implement 
the Carl D Perkins Vocational 
Educatbn Act 


E H White, 
Washington, D C 


1985 $160,000 


8/23/85-8/22/86 New data 
collection 


Vocational 
education 


Provide support services, 
including data collection and 
analysis pertinent to 
department policy issues 


Advanced Technology, 
McLean, Va 


1985 $261 ,827 


10/1/82-9/30/85 Analysis and 
support 


Department policy 
issues 


Study of recent trends in the 
Vocation Rehabilitation 
Program's caseload and 
placement patterns 


Ecosometrics, 
Bethesda, Md 


1985 $81 ,000 
1984 $534,000 


9/1/84-1/30/87 New data 
collection 


Rehabilitation 


Identify and analyze factors 
contributing to the rapid growth 
of proprietary rehabilitation 
services 


Berkeley Planning 
Associates, Berkeley, 
Calif 


1985 $438,795 


7/1/85-6/30/87 New data 
collection 


Rehabilitation 


Putxhase proprietary data on 
freshman college students for 
higher educaiion research 
survey on fall enrollments 


HERI, University of 
California, Los Angeles 


1985 $209,715 
1984 $138,650 


6/29/81-4/1/87 New data 
collection 


College enrollment 


Provide the department, from 
the higher education panel, 
policy-relevant quick-response 
surveys from a sample of 
institutions of higher ed jcation 


American Council on 
Education, 
Washington. DC 
(funds transfer to NSF) 


1985 $140,000 
1984 $130,000 


10/1/82-9/30/86 New data 
collection 


Higher education 


Provide the department 
secondary data collection 3nd 
quick-response analytical 
capability for policy budget ar d 
planning 


Applied Systems 
Institute, Washington, 
DC 


1985 $300,000 


4/1/83-3/31/86 Analysis and 
support 


Higher education 
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Rjuidom Sftmpleg of NIE and OPBE Activities 
for FiMftl Yeais 1980 and 1986 



Activity 


Recipient 


Fiscal year cost 


by award Type 


Study area 


Analyze and synthesize, 
through the Education Analysis 
Center, pertinent past and 
current research and evaluation 
studies; analyze existing 
relevant and complex data 
bases; develop models, 
conduct case studies, and 
perform literature searches and 
reviews 


Pelavin Associates, 
Washington, D C 


1985 $340,631 


10/1/82-9/30/85 An3lys;s and 

CI l^^^^^tl 

buppul I 


Departmental 
policy issues 


Analyze data and provide 
technical support for on-cati 
processing and education 
analysis capability 


Decision Resources 
Corp , Washington. 
DC 


198S $500,000 


7/1/83-12/31/85 Analysis and 
support 


Departmental 
policy issues 


Describe and survey 
longitudinally immersion 
programs for bilingual students 


SRA Technologies. 
Mountain View, Catif 


1985 $725,000 
1984 $500,000 


10/1/83-9/30/88 New data 
collection 


Bilingual 


ouivcy me aiiiiuues anu 
education preference of parents 
of several groups of children 
speaking minonty languages, 
link . ig the sample to NAEP so 
that parental attitudes can be 
related to educational progress 


Educational Testing 
Services. Princeton. 
NJ 


1 yob ^oUU.tXXJ 


9/30/85-12/30/86 New data 
collection 


Language minority 


Analyze the theoretical and 
public policy roots of oenefit- 
cost analysis in rehabilitation, 
examine supplements co the R- 
300/91 1 data base, write 
proposals to develop practical 
plains, based on existing data, 
for models of benefit/cost 
analysis 


Rutgers University. 
New Brunswick, 


1984 $170,920 


9/25/84-9/30/85 Analysis 


Rehabilitation 



^All activities were funded by contract Activities associated with a particular fiscal year often received 
funding in other fiscal years as well Fifteen activities were randomly selected from a list of 26 items 
shown as funded or ongoing in fiscal year 1985 in the annual evaluation report for fiscal year 1985 
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Educational Survey Changes and Proposed 
Information Collection 



Tablt 11.1; Primary, Elementery, and Stcondary Education Surveys and Changes 1980-83 



Survey 


Frequency 


Description 


Change 


Preprimary school 
enrollment 


Biennial 


Sample of children 3-5 years old, enrolled and 
not enrolled by age. sex, race, Spanish origin, 
region, educational levels of parents, 
employment status of mother, household 
income, level of enrollment (nursery school or 
kindergarten), public or nonpublic school 
status and length of attendance daily (all or 
part of a day) 


Temporarily shifted to annual cycle 


r uuiio cici 1 ici 1 loi y aiiu 

secondary schools 


Anniial ^^^^r^ fall ciiruau 

Mnnuai Vmtv/U loll bill vcy 


i>iumucr UT scnooi □isincis, pupiis, sian, ana 
high school graduates and financial receipts, 
expenditures, and teachers' salaries, includes 
the 20 largest cities and outlying areas 


VM^riangeu in luoc lu csiimaies ui iinanuiai 
data and salaries, eliminated in 1983 


Nonpublic elementary 
and secondary schools 


Periodic 


All schools and number of pupils and 
teachers; finances by school affiliation and 
grade 


Noncomparable survey was conducted in 
subsequent year, no other surveys 
appear to have been conducted 


State school system 


Biennial 


Organization, staff, pupils, and finance of 
public schools in the 50 states, the District of 
Columbia, and extra- continental areas, 
selected historical trends 


Appears to have been eliminated 


Offerings and 
enrollments in 
secondary schools 


Occasional 


Courses offered and number of pupils in a 
sample of high schools 


Planned fcr 1983 and 1985 but no record 
of Its beinc conducted 


Revenues and 
expenditures 


Annual CCD 


Current expenditures of school disUicts for 
major functional categories, revenues, capital 
outlay, and debt services ficures also 
included; covers universe of school districts 
for most years 


Increased and more comprehensive 
coverage required for administration of 
department programs 



State education agency Annual Revenues by source and expenditures and Appears to have been delayed or 

operations employees by agency function, including eliminated 

approval of programs, consultative services, 

distribution of resources, general 

management, internal services, planning and 

research, op)eration of schools, and vocational 

rehabilitation; data are reported for states 

grouped by public school membership 



State public school 
finance profiles 


Biennial 


Pupils, teachers, and other characteristics of 
all school districts' resource allocation 
procedures, measures of wealth, and costs of 
education services; program characteristics in 
relation to pupil need and financial capabilities 


Planned for 1983 and 1985 but not 
conducted 


Local school districts 


Annual CCD 


Names, counties, states, principal 
administrators, number of schools, grad** 
spans, enrollments 


Dropped pnncipal administrator listing 


State education 
agencies 


Annual 


Names, titles, and phone numbers of 
education officials for each state 


Changed to periodic in 1982 but not 
listed in 1983 


Number of persons with 
limited proficiency in 
English 


Occasional 


Estimates of the number of children and 
adults with limited English proficiency by age, 
residence, language, famiiy income, and 
country of origin; projections of the number 
for the next 5, 10, 15, and 20 years 


Planned for 1983 but not listed in 1983 



(continued) 
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Survey 


Frequency 


Description 


Change 


Teachers' qualifications 
in second language 


Occasional 


Estimates of the number qualified or partially 
qualified to teach in languages other than 
English and English as a second language 
and the number in projects supported by 
ESEA title VII, the Bilingual Education Act 


Planned for 1981 but not listed in later 
years 


Teacher and 
administrator supply 
and demand 


Annual to mandate 


Estimates of the number of college graduates 
added to the supply, local education agency 
sample of numbers of teacher*^ employed and 
laid off, teaching openings and current and 
anticipated shortages by level and field of 
instruction 


More detail added tn 1981 , changed to 
periodic in 1982 


Student and staff 
characteristics in local 
school districts 


Periodic 


NOES and Bureau of ihe Census collection of 
data on receipts by source and expenditures 
by purpose fro^| state education agencies for 
the entire universe of school districts or 
samples, depending on need; universe data 
are available for fiscal years 1978-79 


Changed to annual in 1982 



1980 decennial school Mandated Tabulations of the 1980 decennial census for Not applicable 

data all 16,000 U.S. local school districts, includina 

population, population characteristics, 
housing, and so on within school district 
t^oundanes c.< data tapes for public use. a set 
;f maps showing the 1980 Census geography 
and school-district boundaries, a geographic 
cross-reference file that identifies each 1980 
Census geographic unit completely or 
partially enclosed by school district 
boundaries, and technical information needed 
by users of the maps and data tapes, the 
product will also include data on sampling 
errors and other information needed for 
extended analysis 



Source Adapted from The Condition of Education , part 2, Programs and Plans (Washington, D C U S 
Government Printing Office, 1980-83) 



98 



Pftge 97 



GAO/ PEMIVSS*'^ The Production and Quality- of Education Information 



Appendix n 

Edncfttional Survey Changes and Proposed 
Inf onnation Collection 



Tabto IL2: Propofd Eimenttiry and Secondry Information Paf System 



FIM 


Area 


Data 


Universe 


Public school districts 


A school district census identification and type 




Public and private schools 


A census of all public and private schools' 




identification, enrollment, staffing, and type 




State aggregate fiscal data 


Revenues, expenditures, and average daily 




attendance 




State aggregate nonfiscal data 


High school graduates, enrollment by grade and 




instructional and noninstructional staff 




Early estimates 


New to the system, universe component 


Sample 


Public school districts 




Public and private schools 


Public and private school teachers 


Public and private school libraries 




Public and private school administrators, new to 






system 




Parents of NAEP students, new to system 


Student performance (NAEP) 


Student progress over time (longitudinal studies) 


Public school finance, under development 


Teacher leavers, new to system 


Policies and practices, new to system 




Early estimates, new to system (sample 






component) 
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Descriptive Data on NAEP and CCD 



Trtbto NAEP FutHflng Hittory 1964>8y 

Fund for th« New 1972 

CamMio AdvancMient Ford lodoral constant 



Fiscal yMr 


Corporatibn 


of Education 


Foundation 


fund* 


Ottieio 


Total 


dollars 


1964 


$112,500 










$112,500 




1965 


260,000 










260,000 




1966 


70,400 


$496,000 








566,400 




1967 


640,000 


640,000 








1,280,000 




1968 


1,000,000 






$372,000 




1,372,358 




1969 


350,000 




$560,000 


1,000,000 




1,910,000 




1970 


350,000 






2,400,000 




2.740,000 




1971 








4,500,000 




4,500,000 




1972 








6,000,000 




6.000,000 


$6,000,000 


1973 








6,000.000 




6.000,000 


5,671,078 


1974 








5,500,000 




5,000,000 


4,745,470 


1975 








4,630,000 




4,630,530 


3,631,788 


1976 








4,900,000 




4,900,000 


3,640,000 


Transition quarter 










1,500,000 




1,500,000 


1977 








4,600,000 




4,000,000 


3.203.343 


1978 








4,800,000 


$62,416 


4,862,416 


3.141.096 


1979 








3,969,348 


917,667 


4,887,015 


2.965.422 


1960 








3,880,000 


459,197 


4,339,197 


2.342.979 


1981 








3,880,000 


1,154.050 


5,034,050 


2.424.880 


1982 








3,880,000 


740,162 


4,620,162 


2.086.794 


1983 








2,500,000 




2,500,000 


1.077.122 


Transition quarter 










1,380.000 




1.380.000 


1984 








4,345,029- 




4,345,029 


1.801.422 


1985 








5,735,480-' 


271 ,231 


6,006,711 


2.439.769 


Total 


$2,782,900 


$1,136,000 


$560,000 


$75,772,215 


$3,604,723 


$83,855,838 


• 



*NAEP was funded by the Office of Education from 1968 to 1973 with a total $20,272,358. NCES in 1974 
through part of 1979 with a total of $28,406,484. and NIE after March 1979 with a total of $30,634,717 in 
direct NAEP funds and about $1 ,067,41 7 in NAEP-related activities Thus, the total direct cost is 
$76,840,161. not including other direct or indirect awards made to the National Commission of the 
States and ETS for NAEP Iike activties from agencies such as the National Science Foundation and the 
Department of Defense 

^Includes "miscellaneous income," revenues from publication sales, carryover from prior years, funding 
from other agencies through awards to ECS and tTS for NAEP. and the like— for example, $25,358 in 
1979 from the Food and Drug Administration for a label-reading assessment, $434,923 from the Depart- 
ment of Labor for an assessment of the 1981 career and work knowledge of 17-year'Okls out of school, 
and $271 ,231 in 1985 from the National Science Foundation to study higher-order skills 

^Includes $352,024 from NIE for the young adult literacy assessnrtent and $1 13,005 from the Office of 
Bilingual Education and Minonty Languages Affairs for \\\e language minority student assessment 

^Includes $1 ,475,480 from NIE for the young adult literacy assessment. $350,000 from the Office of 
Bilingual Education and Minonty Languages Affairs for the language minonty student assessment, and 
$30,000 for analysis of Education Consolidation and Improvement Act chapter ! data collected by NAEP 

^otal inflated by fiscal year carryover, each year represents that year's resources 
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Tabte III.2: Tasting PMwn for NAEP by ^^^^^^^^^^^^^^^^^^^^^^^^^^^^H^^B^B 

Testing Year 1969-86 Content arsa 1969>70 1970-71 1971-72 1972-73 1973-74 

Science • • 

Writing • • 

Citizenship • 

Reading • 

Literature • 

Music • 

Social studies and citizenship • 

Mathennatics • 
Career and occupational 

development • 
Art 

Basic life skills 

Health 

Energy 

Consumer skills 
Literacy 

Computer competence 



ERLC 



Page 100 



101 



GAO/PGMD^M The Production and Quality of Education Information 



Appendix III 

Descripdvo DiUft on NAEP and CCD 



1974-75 1975-76 1976-77 1977-78 1976-79 1979-60 1960-61* 1961-32 1962-63* 1963-64 1964-65 1965-86 



*No data collected 
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TflM 1114: NAEP Target Groups by ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

TMlingYMr1969-S6 Targt group 1969-70 1970-71 1971-72 1972-73 1973-74 



Age 



9 


• • • • 


13 


• • • • 


17 m school • 


• • • • 


1 7 out of school • 


• • • • 


26-33 


• • • d 


Grade 


4 


8 


11 



TaMo iiU: P— Cflptiow of CCD for F^iosMnti 


iry and SocoiHtary Education 


ConipOfiont* 


Charactoftrtic 


I Public school universe file 


Information on all public elennentary and secondary schools in operation dunng a school 
year by school type, grade span, fall membership, and number of classroom teachers. 
availat>ie at school level 


N. Local education agency universe file 


Information aggregated to the state level for the universe of local education agerKies by 
type of agency and operating, fiscal, and control status 


IIL Local education agency nonfiscal report 


Information aggregated to the state level on local education agencies oy number of 
schools in operation. niemt)ership, and f ull-time-equivalent teachers and other staff 


IV. Public school district finance report 


Data on local education agencies by average daily attendance, source of revenues, and 
expenditures b/ fL.iction 


V. State aggregate nonfiscal report 


Fall school enrollment by grade level. fulMime-equivalent staff by maior category, and 
high school 


VI State aggregate fiscal report 


Annual census of state agencies that provide resources to local education agencies, 
aggregated to the state level, on average daily attendance, school district revenues by 
source. ar>d expenditures by major function 



•All components consist of data from an annual census of state education agencies 
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1974-75 


1t75-78 1976-77 


1977-78 1978-79 1979-80 19e0-81« 


19S1-<2 19e2-83* 19e3-M 1984^ 


19IS-M 




• 


b 


c • • 


• • 


• 


• 


• 




• • 


• 








* • 


• 


• 


• 


• 








• • 




• 












• 


• 








• 


• 








• 


• 



*NodatacoNected 

''Social studies and citizenship onty, 

^^Mathematics only 

^k^eer and occupational devek)pment only 
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UNITED 'TATES DEPARTMENT OF EDUCATION 



01 1 ICY 01 IHY ASSISTANT SJCRFTARY 
I OR tDUCATIONAL RFSbARCH AND IMPROVF Ml NT 



HAY I 5 B87 



Mr. Richard I. Fogel 
Assistant Caiptroller General 
Hunan Resources Division 
General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Fogel: 

Thank you for the opportunity to review the draft GAO report, "Education 
Information: Changes In Funds and Priorities Have Affected Production 
and Quality." In general, we agree with the findings cited In the 
report. We also believe that the final report will perform a valuable 
function by documenting a long-term decline In resources appropriated for 
education research, statistics and evaluation, ke did, however, identify 
some major misconceptions and errors In fact. These are discussed in the 
enclosed document. 



Sincerely yours. 




Chester E. Finn, Jr.V 
Assistant Secretary and 
Counselor to the Secretary 



Enclosure 
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n.S.Ptpartnent of Education's 



RESPONSE TO THE GAO DRAFT REPORT, 



f(i 



Education Infornstlon: Changes In Funds and Pr'*orltlc8 



Have Affected Production and Quality' 



Ve read vlth great Interest snd sgree vlth vuc^ of GAO's analysis of the 
decline In appropriation:* for education Information. Vc acknowledge the 
vslldlty of nany points rslsed by GAO for the period covered by the 
report. Since esrly 1985» however, the Departneni hss tsVen clear and 
decisive sctlon to address vost of the problcns cited In the report* 
Vhst follows are our comenta on the report , aa well aa a description of 
Ihe current atate of affalra within the Department* Since In many caaea 
algnlf leant poaltlve changea have been made, we atrongly auggeat that 
GAO's final report acknowledge and describe thoae changea. Else even a 
report that la generally accurate with regard to the period to which It 
appllea may be sorely Inaccurate with regard to the preaent altuatlon. 



RE0RGAKI2ATI0K OP OERI 

Secretary Bennett 'a concern with the quality of the Department 'a 
reaearch and atatlatlca prograna led to planning for Inprovcmenta 
beginning In the flrat montha of 1985 and ultimately to tf s Flacal Tear 
1986 reorganisation of the Office of Educational Reaearch and Improvement 
(OERI). He realised that the Department*a reaearch and atatlatlca 



OFPICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
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tnroffraM tiad not achieved their full potential. Under Secretary Bennett'a 
leaderahlp, the OCKI reorganization vaa dealgned to Iwprove the quality 
of education reaearch and atatlatiral lnfor«atlon by realigning program 
function*, eatab'*lahlng clearer lines of authority • eatabllahlng quality 
control pTocedurea» and strengthening and expanding OERI'a peer »evlev 
syatev* The 1986 OERI reorganization alao aerved to delineate sore 
clearly the reaponalbllltlea of OERI and thoae of the Office of Planning > 
Budget, and Evaluation (OPBE). 

While basically concur vlth the report* a findings aa to the past, ve 
are coneamed thst throughout the report reference Is aade to current 
OERI progm unite by naaet even though the report covers s Period of 
tl«e befotv these units existed. 

Spedflcallyt the "Office of Research" (OR), which vaa created by the 



report, ^t, as a result of that reorganization, the functlona and 
reaponalVQltles of OR are far sore predaely and narrowly defined than 
thoss of "Sfc* National Institute of Education hsd been. GAO'a preaentatlon 
would te Mbatantlally clearer and vore meaningful If the report uaed 
the agency dealgnatlon HIE for the period covered by this study, 1980 to 
1985. In addition* the present tense should not be used when spesklng 
of events tkat occurred during the period 1960 to 1983. 

CHANGES m PRIORITIES 

GAO found ^rlorltlea ahtfted froa new data collection to aeivlce-orlented 
activities such aa dlaaealtiatlon* ao wich ao that the availability of 



See comment 1 . 



Flscsl Taar 1986 OERI reorgsnlzstlon Is s najor focua of the draft 
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up-to-date Information to disseminate to teachers and other practitioners 



Issue as one deserving analvsls, we think that the assertion made In the 
report Is rot well founded and should be placed In context. Throughout 
the report, GAO describes dissemination as a service activity, quite 
distinct from research activity. While this often can be the case, the 
relationship between research and the dissemination of research lies at 
the heart of our work In OEKI. It should be recognized that a typical 
basic research report seldom Is an effective mechanism for Impact on 
education practice. It must often be re- Interpreted and synthesized to 
permit effective use by teachers and others It Is Intended to benefit. 
In pre-NIE days, funded researchers tended to see their end products as 
reports or journal articles for other scholars. As a consequence, NIE 
received complaints from policymakers and practitioners that such 
reports were practically useless to them. The claim was made that 
researchers had an obligation to make their results Intelligible to 
those concerned with Immediate Issues of education practice. With 
encouragement from the National Council on Educational Research, NIE 
Instituted In 1980 a requirement that investigators indicate how they 
planned to address the dissemination function. Slmll ar requirements 
were placed on the R&D Centers supported by NIE and. In particular, on 
regional laboratories, which have a special responsibility for 
dissemination. In many cases, such activities were already planned and 
being carried out, but now they became explicitly characterized as 
"dissemination.** While It Is true that greater emphasis hjs been placed 
on dissemination, we do not agree that this emphasis alone poses any 
threat to new information production. 



Now page 2 



mav be threatened** (Executive Summary p. 3). Although we acknowledge the 




ERLC 



Page 107 



GAO/PEMDW4 Hie Prodnctloii and Qiwllty of Education InfonnatlQii 



Appendix iV 

Comments From the Department 
of Edncfttion 



Now page 83. 



rAO ItrplleB that changes In leadership necessarily result In chinp.es In 
priorities (p. 117). The single example cited bv GAO Is a unique event, 
and surely Is not sufficient to prove GAO's point. In fact It la more 
correct to assert that despite changes In leadership, there has been a 
significant degree of consistency and continuity In research priorities. 
This consistency will be reinforced by the provisions of the FY 1987 
reauthorizing legislation for OERI , which r^^ulres the Secretary to 
publish reses'-ch priorities every two years. 



Furthermore, as OERI budget levels declined over the years, funds were 
not available to permit significant changes In priorities since an 
Increaslrs percentage of the OERI budget was being restricted by the 
Congress for Institutional awards. 



Now page 20 



NEW DATA COLLECTION 

GAO found that many areas are no longer being studied, at least with 
respect to federal support for new Inquiry, and that they have not been 
replaced by other areas. Somewhat cryptically, the report states t'nat 
"New data collection appears to be psrtlcularly out of step with areas 
Identified by education experts as those In need of educational reform" 
(p. 19). 

It should be noted that a significant portion of OERI-sponsored research 
Is conducted by the national R&D Centers. In December 1985, awards were 
made for the establishment of ten R&D Centers. These awards were made 
following a nation -^Ide competition which had been preceded by an 
Intensive two yaar planning effort characterized by Involvement of 
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iiitnerou8 and varied publics. As a 
reflects nany of the nost Important 
today » particularly those pertinent 
movement In this country. 



result » the work of the Canters 
questions snd Issues In educstlon 
to the current educst Ion reform 



Now page 34. 



Mention Is made by GAO of s report prepsred for the Nstlonsl Council mi 
Educational Resesrch (p. 33). This report Identifies "ths nost crltlcsl 
areas" for resesrch (e.g. Improved tescher preparstlon* strengthened 
curriculum In math» aclence> etc.). GAO asserts In Its drsft rsport 
thst "there were few awards for new dsta collection on theac toplca In 
1985". While this msy be true for the period covered by the GAO study* 
In April 1987» DERI announced Its Intention to sponsor four Mlnl-Centsrs 
snd Invited spp Heat Ions. Applications were Invited for new 
Mini-Centers focusing on lesmlng and teaching In selected content 
areas: (1) elementary education; (2) mathematlca; (3) literature; and 
(4) science. They will represent a much needed coord, ^ted effort Co 
examine teaching and learning In some of the scademlc subjects thst 
comprise the core school curriculum. 



DERI slso hss been more crestlve In using Its declining resources to 
address critical problem areas. For example: 



* For the past seversl yesrs» OERI has supported sctlvltlss of sn 
Urbsn Superintendents* Network. This yssr ths Network Is 
advising OERI on the prspsrstlon of s publlcstlon on school 
dropouts thst will focus on programs » pollclss* and prsctlcss 
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that Network nenbers have found to be effective In deallnf^ with 
"students at risk"; 

* An Intramural research project, currently underway In OR, 
examines the effects of high poverty schools on the pupils 
attending them. The project looks at Issues related to school 
climate, school policies, and school resources so as to 
understand more precisely what makes a dlffere^ice In Improving 
educational opportunities for the poorest students; and 

* OERI's FY 1987 research grants on reading and literacy are 
supportlrg studies of adult literacy and reading achlevetrent of 
students from low-socioeconomic backgrounds. 



Now page 20 



PECLTNK IN SUPPORT FOR IKDTVTPUAL RESEARCHERS 

A shift from support for Individual researchers to support for 
laboratories and centers In the Office of Research Is cited In the draft 
report (p. 19). It should be recognized that over the years the Congress 
has directed ever larger portions of the OERI budget toward awards to 
Institutions (e.g., R&D Centers, regional educational laboratories, 
NAEP, etc.). For example, under congressional directive, A7t of NIE's 
FT 1981 funds were awarded to the 17 regional educational laboratories 
and the R&D centers. In FY 1987, these same Institutions received 61 1 
of OERI's total appropriation and all OERI Institutional awards 
accounted for 79%. As a consequence of congressional action, there has 
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been a steady ''ecllne in the amount of flexibility OERI has had to make 
Awards to various types of "information producers." 

Despite scarce resourr^s, however, OERI already had set aside a portion 
of its budget for a field initiated grants competition in FY 1986. The 
Field Initiated Studies Pro ratn of the Office of Research was designed 
to generate proposals from Individual researchers on topics of importance 
to education Improvement. As a result of that competition, ten grant 
awards were made for an FY 1986 total of $724,248. Subsequently, in 
reauthorizing OERI in FY 1987, the Congress mandated a minimum annual 
funding level of of $500,000 for field initiated grants. Currently, 
applications are being invited for a FY 1987 Field Initiated Studies 
Program. 

Also in FY 1986, the OERI Fellows Program was established to further 
stimulate interest among individual researchers and scholars in education 
issues and topics. Seven researchers, scholars, and practitionevs apent 
up to one year in OERI conducting independent research on topics ranging 
from student financial aid to school productivity. Applications are now 
being accepted for the FY 1987 Fellows Program. We expect that three to 
five individuals will receive OERI fellowships. 

CENTER FOR EPrCATIOW STATISTICS 

Secretsry Bennett's continuing interest and commitment to repairing the 
national data base on education is a matter of public record. For the 
second consecutive year, the Department has requested a substantial 
Increase in the CES annual appropriation. Congrees denied that request 
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In ry 1987. In FY 1988, the budget request for CES represents a 53% 
Increase. 



In response to questions about the timeliness, quality, and relevance of 
Center for Education Statistics* (CES) data, the Department has : 



(1) undertaken the sweeping "redesign" project mentioned in the 
report; 



(2) supported work of the State Education Assessment Center also 
mentioned In the report; 



(3) organized the Study Group on National Assessment which reported 
In March 1987 with dramatic recommendations for the future of 
the National Assessment of Education Frogreas (NAHP); 



(4) provided significant additional staff resources; and 



(5) proposed, as part of the Administration's legislative 

anendments for the Education Consolidation and Improvement Act, 
authority to create a new Cooperative Education Statistics 
System that will eatsbllsh joint Federal/State efforta to 
improve the quality of education atatlstlcs. 



(6) made public, for the flrat time, a schedule of publlcaclons 
for the flacal year. Of the 94 publications scheduled for 
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release during the first six months of FY 1987, 90 have been 
released t and five others scheduled for later release also 
have been published* This exceeds the total number of 
publications released In all of FY 1986. 

Vhlle It will take a great deal of effort and time, as veil as more 
favorable congressional action on our request for additional resources, 
ve believe that the course we have embarked upon will lead to data that 
are timely, reliable, and comparable across States and localities — and 
responsive to the criticisms found In the GAO report* 



See page 45 



See comment 2 



Elementary/Secondary Data Collection 

GAO acknowledges on p. 55 of the draft report that data on elementary 
and secondary education are gathered from several distinct survevs. 
However » we are concerned that the narrative In this section of the 
report appears to confuse Common Core Data (CCD) with CES' entire 
elementary and secondary data collection system. CCD actually is just 
one of several parts of the whole. The redesign effort looked at the 
full array of elenentary/secondarv data collected through CCD, NAEP, 
longitudinal studies, and the Public and Private School Surveys. 
Several statements In the report Imply, Incorrectly* that (1) CCD was 
the primary elementary/secondary data system and (2) that It was the 
appropriate mechanism for collecting a variety of data. 



CCD was and will continue to be the component of the elementary/secondary 
data system through which basic universe data are obtained for use 
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primarilv as ssnpling fraves for surveys. In addition, essentlsl core 
data thst are needed from every State are and will be 
collected — enrol Iments, diplomas, staff and fiscal data. Hovever • CCD 
has not been and will not be the component through which data on 
educational processes and outcosiea are obtained. 



Now page 38. 



INCREASED USE OF COWTRACTS 

We are particularly concerned about the implication made In ^ . draft 
report that the use of contracta rather than grants is "more likely to 
constrain inquiry" (p. 40). The report atatea: 

"in teraa of the implications for educational information, typically 
contracta involve greater specification of queationa to be 
investigated and atudy design than granta. and thua are more likely 
to constrain Inquiry* Also products of contracts typically are 
reviewed by the funding agency before release while products of 
grsnts typically are required after releaae." 

OERI uaea many procurement methods to accomplish its work. For example, 
grant awarda were made to the R&D Centers while the regional laboratories 
are working under contracts. Typically, contracts are used when OERI 
wants to (1) ensure that the substsnce of the work being procured is 
clearly articulated and (2) provide for an appropriate level of 
accountability. 

Aa a consequence of OERI*s External Advice and Peer Review Policy, the 
eatablishment of priorities aa well as the identification of research 
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questions and topics are accovpllshed with the Involveinent and advice of 
researchers, practitioners, and pollcy«akers external to OERI. However 
prescriptive Its procurevients nay appear, OERI work scopes arc frequently 
shaped with the advice and counsel of outside experts. Additionally, 
vhen OEKI Issues a request for proposals (RFP) , work scopes usually 
contain an Invitation for alternative approaches. Rather than constrain 
Inquiry, OERT Is steadfast In Its attempts to foster creativity and 
Invite alternate strategics and fresh Ideas. 



OFFICE OF PLANNING, BUDGET, AND EVALUATION 
CHANGES SINCE 1985 

The GAO draft report looks at OFBE activities from 1973-1985, with a 
focus on 1980-1985. During this period, although OPBE has been the 
central evaluation office of the Department (and previously the Office 
of Education In DHEW) , program offices have conducted many evaluation 
studies. The resulting Information has often gone to the Congreas and 
Is a significant component of the evaluation Information available to 
it. 

Since 1985, Important changes alao have t •en made by Secretary Bennett 
in the Department's evaluation program. The Office of Planning, Budget, 
and Evaluation (OPBE) has s redefined role. It Is still the central 
evaluation office, but also coordinates planning and Implementation of 
evaluation atudles throughout the Department. 
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Dcpartvcnt-vlde procedures for planning and coordinating evaluation 
studies were catabllahed In Auf.ust 1986. Under these procedures* OPBE's 
responsibility for evaluation ac*'ivif-<f*3 in the Department were consistent 
with the Secretary's atateacnt of July 2* 1983 on reorganizing 0^1 • 



The objectives of OPBE's collaborative efforts with progrsin offices 
conducting evaluations are as follows: 



o Ensuring that evaluation studies address key policy Issues; 
o Ensuring high vethodcloglcal quality in studies; and 
o laprovlng the dissciilnatlon of pol lev-relevant findings and 
their inpllcstlons. 



Results of the first year's sctlvlties under those procedures sre 
encouraglrg. OPBE now has a central Inventory of ongoing evaluation 
studies. There Is also a fiscal year 1987 evslustlon plsn for each 
program area, representing consensus between OPBE and progrsin staffs on 
priority Issues* wethodology* cost* funding sources* and timelines. 



OPBE staff sre reviewing draft work statements and draft study reports 
to help er.jure high quality of analysis snd reporting. OPBE hss 
completed a auiwary report on evaluation activities actually Implemented 
in FY 1987. OPBE also has Issued guidance for the FY 1988 cycle of 
evaluation planning and coordination. 



The condition of evaluation Information in the Department Is significantly 
different in mid-fiscal year 1987 from that of 1985 and preceding years. 
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See comment 3 



See comment 4 




As the procedures for plsnnlng snd coordinstion of evslustlon studies 
become aore flraly estsblished, they vill further inprove the quality of 
evslustlon Infomstlon svsllsble to the Congress. 

COKKECTTONS TN NUMBERS 

Hsny nuobers snd percent sges given for contrscted evslustlons should be 
chsnged. For exsnple, the '*79 percent decline" Is too high. The CAO 
numbers do not tske Into sccount tssk orders to support contrsctors. 
which sre often fo small-scsle studies. Furthemore* In cslculstlng 
dollsr snounts, the Annuel Evslustlon Keport for the yesr sfter the yesr 
reported on should be used (e.g.» the FY 1983 AER hss the best svsllsble 
numbers for FY 1982). This Is prlmsrlly becsuse of the former 15-month 
avsllsblllty of Title T/Chspter T funds. Best svsllsble numbers for 
OPBE obligations, FY 1980 - 1983, sre ss follows: 

FY 1980 - $17.6 million 

1981 - 16.8 

1982 - 9.1 

1983 - 10.9 

1984 - 12.7 

1985 - 9.4 

Ressons for Declines In Numbers 

There vss unquestlonsbly s decline In the number snd size of OPBE 
evsluati. n studies from 1980 to 1985. These Importsnt fsctors contributed 
to the decline: 
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o Many categorical prograaa (e.g. , library support) were candidates 
in Adsinistration proposals for block-grant consolidation or 
elimination. 



o Sose congressional criticism in the 1970*8 focused on the costs of 
large-scale evaluations. Other criticisn said that evaluation 
resulta did not clearly show vhether progress "were working or not 
working." The decline in numbers and costs of atudies was* in 
part, a response to the criticisn. 



o The decline in OPBE's studies of ECXA Title I/Chapter 1 resulted 
largely from the congressional mandate to OERI for a large-scale 
multi-year study of compensatory education. Furthermore, Title 
I/Chapter 1 was itself a candidate for block-grant consolidation in 
the early 1980*8. 



See comment 5. 



Now pages 31*33 



COWFUSIOM OF TERMS 

There is frequent confusion throughout the report between **awards*' and 
''activities." They are not synonyms. "Awards" refers to new contract 
or contract modifications, including task orders. An ongoing "activity" 
may not involve an "award*' in any given fiscal year. The confusion of 
words makes the numbers misleading (e.g., pages 31-32). 
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See comment 6 



ADDITIONAL COMMENTS 



More specific cormBents and. In many Instances, corrections to the text 
are Included on the following pages. In most cases, ve have provided 
the page numbers of the report to which the comment Is relevant. 
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The following are additional gag comments on the May 16, 1987, 
Department cf Education letter. 



1. The Department of Education noted that much of our analysis per- 
tained to units that were in operation prior to the current organization 
of OERL In the draft, we acknowledged the historical context but 
referred to units by their current names. The department suggested that 
for clarity and accuracy we refer to units by the names that were used 
during the period of our review. We concur and have changed the report 
accordingly. 

2. The department expressed some concern that readers may think that 
the (Common Core of Data is the only data that ces collects for elemen- 
tary and secondary education, although in our discussion, we noted that 
it is one part of the elementary and secondary education data collection 
system. We chose our example because it represents one of several gen- 
eral ways of gathering data. We did not state that it was a primary 
mechanism, nor did we state that it is an appropriate mechanism for 
collecting a variety of data. We have made several minor changes to 
remind readers that administrative records represent one of several 
ways in which data can be gathered. 

3. The department stated that our estimate of the decline in evaluation 
funds was too large because of the procedures we used to derive yearly 
funding levels. In determining funding levels for opbe contracts, we used 
the procedures the department described in its comments. We limited 
our analysis to contract activities, because those were the only evalua- 
tion funds documented in the annual evaluation report. Time did not 
permit our verifying the updated figures provided by the department, so 
we have not included them. However, the department's numbers do not 
appreciably alter our principal findings. 

4. We agree that there were many reasons for the observed declines in 
evaluation activity. The department's are likely explanations. We 
offered others in Federal Evaluation: Fewer Units, Reduced Resource, 
Different Studies From 1980 , gao/pemd-97-9 (Washington, D.C.: January 
1987) — namely, shifts in emphasis toward management-oriented stud- 
ies, internal evaluations, and low-cost studies prepared primarily for top 
agency officials. The data in our report did not permit us to assess the 
relative importance of each reason. 
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5. The department correctly noted that '^awards" and ''activities'* are 
not synonyms. We were limited to using available documentation in con- 
structing our data bases for analyzing changes in activities and awards 
over time, but we have attempted to clarify this distinction. 

6. The pages of additional conunents from the department have not been 
reproduced in this appendix. However, where appropriate, we consid- 
ered and used its specific conunents and corrections in preparing this 
report. 
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